The Top 10 (or so) LastDropofInk Articles of 2011

/by

In this article I recall my top 10 favourite and most notable articles of 2011 and provide a reflection on each of them.

2011 saw myself and the LastDropofInk site go through some major changes, the site got a complete redesign in August and that dead mosquito was finally laid to rest and a new shiny design was put into place. This also saw me step into the world of video too and my thanks to AndyLuckyShot for helping me with the introduction section to the video content, the comments that you all made around the quality of them and I also took the opportunity to explore some of the topics that had been on my to-do list for quite some time.

A quick count shows that I created over 100 articles for the year, for which I’m pleasantly surprised, as some weeks I’d add nothing and some weeks 5 or more articles in one hit. I openly admit that some of them were weak, however there are several that helped me (and hopefully you too) understand some of the complexities that impact the way we work and the way that the marketplaces also work too.

Top 10 Articles of 2011

These are my top 10 articles of 2011. There are a lot more than just 10 though, as many of them are interlinked or were created in parts that lead on from each other. This is why its easy for me to reflect on what was documented over the year as I can see the full picture now of where each article was going and how they were linked together.

#1 – Tesco Marketplace

Tesco LogoIn November 2010 we heard concrete rumours that Tesco were going to launch into the marketplace arena, the launch date kept slipping throughout 2011 and now its set for sometime 2012.

However that didn’t stop me pondering what positives could be taken from the existing two key players in the marketplace arena, eBay & Amazon and from niche sites like Etsy and to ponder what would happen if you could rewrite the marketplace platform from scratch.

The article was titled “Part 1: Battle of The Giants – Tesco V’s Amazon – Who Will Win?” and it quickly exceeded 25,000 words so I published the first part in January.

If you look through the comments you’ll see that I didn’t actually publish the latter parts of articles detailing the steps and USP’s that I felt that were needed to create a killer marketplace. After all, Tesco in my mind is the only company that I know of that could actually take on Amazon in the UK and make a serious dent.

I did get my chance to pass a few of these ideas over earlier this year, but frankly screwed the chance up as the person I spoke to, didn’t really comprehend what I was discussing and it was entirely my fault for not putting it into a language that they understood.

When Tesco finally launch the marketplace, it has the potential to make a minor dent to Amazon’s trade, but sadly I suspect it will not be the marketplace that so many businesses are looking for to provide viable alternatives to the major players eBay & Amazon.

#2 – SEO for eBay – 10 Minutes Per Day

elephant-car

See even elephants need spoonful sized mouthfulls too. Yummy

While this article only scratched the surface, the important note was made.

That being that even with 10 minutes per day, the cumulative effect of lots of minor changes over time would become a huge positive.

The article titled “How to: SEO for eBay – 10 Minutes Per Day” first tackled the perception that it’s a large job, but a spoonful at time was perfectly manageable and then moved into the 5 key areas to focus upon.

#3 – Making Sense of the Pending eBay UK Updates in May 2011

girl-cryingAfter deciding to take a different stance towards the updates that eBay released in May 2001 (frankly, I had been too negative in some cases), I included my favourite image and quotation to date.

As far as I see it there are two options
#1 You cry like a little girl
#2 You fully embrace the changes in advance

There are two types of sellers on eBay, serious merchants and ones that chuck their toys out of the pram and cry at every turn. This was a prime opportunity for me to poke fun at the latter and help push the former seller types forwards with information that they could actually use to embrace the changes.

In these updates we saw the fee structure change for eBay, mainly for the better, multi-variations go live in many more categories and also the reincarnation of the failed eBay express checkout, adapted to a shopping cart and what we see now on eBay UK today.

#4 – eSellerPro (x2)

eSellerProI can tell you neither of these won me any browny-points, but as far as I am concerned they needed to be written and I’ll explain why now. This is actually a combination of two articles, the first is “The Unofficial History of eSellerPro” and the second being “Are There Alternatives to eSellerPro?“.

In the unofficial history of eSellerPro article, I pointed out the incubation factors that led to the creation of eSellerPro. There were a couple of factors, but the two main ones were the failing of both MartketWorks and ChannelAdvisor to adapt to the UK market fast enough.

The incubation period didn’t just spawn one new product/service provider (that being eSellerPro), it actually spawned at least one more within a 2-3 month window. Sadly [for eSellerPro, which was a product that I personally worked on to solve, not hinder merchants], with hindsight at my side, I now understand that there are specific conditions right now, that will spawn a 3rd generation take (they’re different, but they’re there), for which we’ll probably see arrive publicly by late 2012.

To add to this statement, I’m not part of this, but I am open to offers as CEO or evangelist. You can contact me directly here to discuss further.

This feeds nicely into the second part of this section, as when I wrote the “Are There Alternatives to eSellerPro?” the whole point was that you[as a merchant] should not change providers at all, instead try and look at the provider differently and solve the issues  that were hindering your business differently.

#5 – Affiliate Networks

Again this is a combination of articles, all focused around affiliate networks as an additional sales channel . It’s a topic that I am looking forward to expanding in 2012 and I will be posting more on in 2012.

  1. Part 1: What Are Affiliate Networks & Should I be using Them?
  2. Part 2: What Are Affiliate Networks & Should I be using Them?
  3. The Top UK Affiliate Networks

Affiliate networks can offer a new sales channel for medium (and larger) sized businesses and not just larger corporates. As the entire channel is pretty much working on “commission  on confirmed sale”, this is very similar to the eBay & Amazon marketplaces, just that it takes longer and the overheads are different.

To keep this as easy as possible for you:

If you’re paying over £300 a month to eBay or Amazon and have a website that is half decent, then affiliate networks should be seriously considered.

#6 – Using eBay Shop Keywords to Leverage the Extra eBay Shop Pages

eBay Shop keywordsThis was a fab article for myself (and you hopefully), as I had been meaning to complete this for months.

After finally sitting down and taking the time to document the eBay Shop  keywords that can be used in the custom pages of eBay shops and included plenty of examples, it turned out to be a thorough article and also lead to the article I was aiming to do which was another titled “How to Make Your Own eBay Daily Deals & Weekly Deals”

#7 – The Book Responsible for Where I am Now

“I owe everything to this book and if I can learn its contents in a week, in a caravan with no computer, you can too.”

I can actually tie the reason why I am where I am today down to a single book. There’s no need to explain this further as I made a video especially for this which is below:


Video not showing? Click here

#8 – The E, A, S & Y Model

In 2011 I was able to hammer out an approach that was simple to follow and would allow me to take a structured approach that both myself and clients could easily work to, helping them with their businesses.

I wanted this to be as easy as possible so I named it as “EASY”. The crux of this model is outlined in three articles, these are:

  1. E is for Efficiency
  2. A is for Add more Sales Channels
  3. S is for Stock
  4. Y is for why or who is doing it

You’ll notice that there is no dedicated article for the Y part, this is simply because its not a speciality of mine and as such I have drawn the line at this point. The first 3 stages will impact the fourth stage, however this stage requires skills and experience that I just don’t have and if you have the first 3 stages licked then your game is huge!

#9 – The Limits of Marketplaces

Why do eBay Sales Stay ConsistentI had first wrapped this up as a ‘conspiracy theory’ in late 2010, but actually that was bollocks.

Marketplaces such as eBay & Amazon have limits, they were once thought of as having unlimited growth and merchants shouldn’t have any reason at all to stray from just one or both of them, because all the customers they could possible reach were there.

As 3rd party software as developed and evolved over time, its allowed businesses to sell over multiple channels and even on the same channels but under different persona’s. Reaching a wider and wider audience. We’ve seen this in reverse with larger corporates (see this article) now leveraging eBay & Amazon in a trend which is likely to peak in 2012, as they realise that these marketplaces are viable, complementary channels for their existing operations.

Such limitations were first covered in the article “Why do eBay Sales Stay Consistent?” in September 2011 and I am now 100% sure such a modifier that I discussed in second the article “The eBay “Best Match” Position Bias Modifier Hypothesis” is actually true. Its true because with the unique view that I have had over the past few years as proven it to be the case and ironically its exactly what I would do!

#10 – UnderstandingE, code-named “ProjectE”

understandingeAfter flirting with this idea for years, it wasn’t until someone approached me with a similar idea is when the wheels started to turn on the original idea earlier this year.

I had the huge pleasure of working with 6 people (not businesses) after making a rather public and free commitment to 5 businesses in August for eight weeks. The first video of me making the commitment is here and the second video with a more formal invitation is here.

It was only meant to be 5 businesses that were included, there were +50 applications that I received and a sixth snuck-in. Ironically, it was the 6th that had the largest increase to their business post the eight weeks with growths of hundreds of percent and are now posed with operational issues that the other 5 had hit and overcome some time before. What a “good problem”.

This allowed me the opportunity to play with a few ideas I had and test them out, outside of my normal client base. The outcome wasn’t exactly what I expected, I discovered a new factor that I must include in the UnderstandingE business when its launched in January.

I’d like to take the opportunity to thank Jason, Gill, John, Paul, Sue & Danny for allowing me an insight into your businesses. What you allowed me to do was absolutely fundamental to the pending launch of UnderstandingE.

Summary

I know I’m supposed to summarise here, but one last article that is worth of a mention is an article I wrote about the Podcasts I listen to and one of my idols posted a comment back here on the site, the article and comments are here. This had everything to do with Dave and thank you, it made my year!

I’d like to thank you for taking the time to read the articles I’ve written, I’m hopeful that you have found them exceptionally useful.

What I have been able to do is create content that is timeless and while flirting with the UnderstandingE project and procrastinating on it heavily for months, I’m ending this article on the note that UnderstandingE is a go! And as such you’ll be hearing &amp seeing me a lot more in 2012.

3% of Sales Coming from the EU? Say Hello to the EU VAT Bomb

/by

If your business has been happily taking orders from around the EU and even if this is as low as 3% to one EU country hot-spot for a £1M turn over business, then you could be sat on an awkward issue regarding VAT.

This was another one of those ‘ah-ha moments’, although quickly followed with ‘wtf’ and ‘omg’ that’s going to be fun. In this article I do my best (as I’m not an accountant) to bring this topic into plain English.

This is certainly an issue that hasn’t been highlighted to businesses from either Amazon or eBay to my knowledge and is quite concerning eBay has been in such a push for Cross Board Trade (CBT) in recent months with statements that it fuels half a billion export boom and pushing the international site visibility upgrades and featuring eBay UK listings on the EU sites.

Disclaimer

Firstly lets clear a few points:

  • #1 I’m not an accountant
  • #2 I’m not a VAT expert
  • #3 I’ll never be #1 or #2.
  • #4 If you are #1 or #2 or have experience of this topic with your business, then fell free to comment and I’ll amend this article accordingly as needed.

That said let’s dig in and take a look a potentially nasty issue for medium to larger eCommerce businesses, because its really easy to go past the VAT limits, as they’re sooo low!

The UK VAT System

Starting from the beginning, VAT in the UK is pretty much a given. At around £73K of gross turnover or if expected to exceed this, a business is pretty much required to register to become a VAT tax collector and give the HMRC 20% of sales (and to be able to claim back the VAT on purchases and on expenses). This may not be technically correct, as there are deviants of this, but that’s the crux of it for most product based businesses.

So if you exceed or ‘expect to’ exceed ~ £73,000 of sales in one year, you normally need to register for VAT in the UK. Relatively straight forward so far. Yay!

The EU VAT Thresholds

Now this is where it can get nasty. As that expectation to exceed the threshold in the UK, also applies to business delivering products from the UK into the EU member states (called distant selling), but these states don’t all have such high thresholds when compared to the UK. In fact some of them are really low at around £30,000!

Let’s say you’ve been happily running a £1M p/a business that uses the marketplaces and also a couple of other sales channels. As you’ve been processing your orders from the UK, in most circumstances you’ve also bee taking orders from a wide variety of countries outside of the UK too.

The shocking reality is that if as little as 3% of these orders have come from one of the EU countries such as Austria, Belgium, Cyprus, Estonia, Finland, Greece, Ireland, Malta, Portugal or Sweden. You have an issue. 

Each of these countries have a VAT threshold of €35,000 (approximately £30K), when you hit this threshold then apparently you’re then required to register for VAT in those countries locally.

VAT Thresholds For European Countries (Sept 2011)

Doing some homework prior to writing this article, I found this document on the European Commissions website under Taxation & Customs Union. The middle section shows the thresholds for each of the member states in this document and I’ve extracted this to form the table below and highlighted the thresholds in bold.

Member State Threshold for application of the special scheme for distance selling
Belgium €35.000
Bulgaria 70.000 BGN (€35.791)
Czech Republic 1.140.000 CZ (€46.570)
Denmark 280.000 DKK (€37.557 )
Germany €100.000
Estonia €35.151 
Ireland €35.000
Greece €35.000
Spain €35.000
France €100.000
Italy €35.000
Cyprus €35.000
Latvia 24.000 LVL (€34.052)
Lithuania 125.000 LTL (€36.203)
Luxembourg €100.000
Hungary 8.800.000 HUF (€32.257)
Malta €35.000
Netherlands €100.000
Austria €35.000
Poland 160.000 PLN (€40.293)
Portugal €35.000
Romania 118.000 RON (€28.012)
Slovenia €35.000
Slovakia €35.000
Finland €35.000
Sweden 320.000 SEK (€36.232)
United Kingdom 70.000 GBP (€81.843)

Even with the approximation that €35,000 is £30,000, for a £1M p/a business, this is only 3% of sales and actually very easy to achieve by sheer accident into one or more of these countries.

It’s reassuring to know that the more likely culprits to top the £30K limit such as Germany, France & Italy have a much higher threshold of €100,000 or approximately £83,000, although in some circumstances, this could also be easily achieved without a second thought.

Summary

The aim of this article was purely to highlight and bring this topic into a ‘plain English’ format. I hope I have done that successfully.

If you even suspect that you’re likely to approach these figures to any of the countries above in your trading year, then please consult expert advice, such as from your accountant.

PS: If anyone knows how this applies to Amazon FBA items that are held in say France or Germany, I’d love to hear from you and please post a comment in the comments section below.

Creating Your Very First iMacro – Web Browser Automation

/by

Following on from my earlier article on How to Install Firefox & The iMacros Addon which also included some performance tips, this video shows you how to create your very first iMacro and also a more complicated macro, that performs a useful action.

In the video I add an extra command called “WAIT”. The wait command allows iMacro’s to pause for a period of time for an action to be performed and as with all commands in iMacro’s the command needs to be in UPPERCASE.

Urgent: Dealing Amazon Pending Orders

/by

 

Firstly apologies, I’ve know about Amazon’s pending orders for a long time now and they can be dangerous. I should have considered their impact at this time of the year earlier. I never suggest anything as ‘urgent’ that I cover here, but this is the exception…

What Are Amazon Pending Orders?

Amazon pending orders, are orders that have one or more issues with either the customer or the payment for the order. This means that there is an order that has been taken by Amazon, but not yet been made available for download and/or processing.

Taking a direct extract from the Amazon help system, pending orders are explained below:

Orders in pending status indicate an issue with the customer’s payment method. Those orders are not in a shippable status, and should not be shipped, even if the buyer contacts you directly. Pending orders will not have confirm and cancel buttons in Manage Orders, and they will not appear in either the Order Report or the Unshipped Orders Report.

Once an order is in a shippable status, the confirm and cancel buttons will appear in Manage Orders and the order will appear in both the Order Report and the Unshipped Orders Report. At that point, the order should be shipped and the shipment confirmed in Manage Orders or with a feed.

Note that Amazon assumes the risk of buyer non-payment for every order in your Unshipped Orders Reports or that has confirm and cancel buttons in Manage Orders. Of course, just as now, there may be circumstances where payment may later need to be refunded, for example when the buyer does not receive the order, or it is not as described.

Why Are Amazon Pending Orders Dangerous?

Using the Amazon help explanation of pending orders, I’ve highlighted the key points above “orders are not in a shippable status” and “will not appear in either the Order Report or the Unshipped Orders Report“. To clarify such orders are dangerous because:

  • They’re sat in a different queue
  • Stock may or may not be allocated to such orders
  • They don’t appear in the standard (or any) reports that can be exported

Pending orders are both a liability & risk to the business. Such orders can sit in this status for weeks sometimes, however at this time when there is a higher velocity of sales than normal the low risk of these turn into a serious risk.

Stock can be locked in Amazon orders and 3rd parties may not know of them, thus keeping the stock available on other channels and open the business to higher chances of overselling. The orders are not shown in the order reports and any system, whether it’s the sellers own or a 3rd party can overlook these and not account for them in the current stock levels.

The Scale of Amazon Pending Orders

Now this is curious, I checked on four Amazon Seller Central accounts this morning, roughly their account types were like this:

Average Order Value Order Velocity
Very High (over £150) Medium
Medium (below £50) High
Medium  (below £30) Medium
Very Low (below £10) High

Now my logic beforehand would have indicated that both the “very high value” and the “high order velocity” for the “very low” orders would have been the two accounts that displayed issues with pending orders.

However I was wrong, both these accounts had very few pending orders (one had 3 and the other 11). It was both medium order value accounts with moderate to high volumes of sales that had the most number of pending orders, one account had over 150 of them and the other was over 50.

However they ALL had pending orders, some were showing back to late November and the numbers differed from account to account (they’re four unrelated accounts in four different product verticals).

How Do You Locate Pending Orders?

Locating Amazon pending orders is relatively straight forwards, there are several paths and I have also included a direct link to them as well:

  • After logging in to your Seller Central account, on the right column, scroll down to the section called “Your Orders (Amazon.ext)”, under the section called “Seller Fulfilled” there is a line for pending orders and the count is a link
  • From the top menu, hover on “Orders” and select “Manage Orders”, then click on “Advanced search” and select a date range (I’d suggest 365 days for the first time) and set the “Order Status” drop down box to “Pending” and hit search
  • A direct link to the last 180 days of Amazon pending orders is here and if you want to search for less or longer change the value of the “preSelectedRange” attribute

How To Handle Pending Orders?

Firstly the good news…

3rd parties such as eSellerPro, ChannelAdvisor & Linnworks use Amazon’s newer API called MWS (Marketplace Web Services).

Why is this important? Included in the calls is the ability to locate and account for pending orders. This allows them (3rd parties) to allocate stock lines that are held in pending orders so that they’re not allocated for sale on other sales channels (like eBay or websites for example). If you’re using either of these then you have nothing to worry about.

This is of course assuming that you have setup MWS with these platforms. If there is any doubt at all, contact them immediately.

Now the bad news…

If you are using the order reports from Amazon to process your orders either manually, through a 3rd party or through AMTU (Amazon Merchant Transport Utility) then you are likely to have a potential issue.

The ability to integrate to MWS is there but requires time & a developer, this is ruled out. The same goes for integrating to any of the fore-mentioned 3rd party platforms. So you’re stuck in a limbo period.

Also this only affects merchants that either use a virtual business model or are also selling the same products on other platforms/marketplaces. If you only sell on Amazon and work in a stocked business model, then you can ignore this completely.

Dealing With Pending Orders Manually

However if your business does sell its product ranges elsewhere or uses parts of a virtual product model, then I suggest the following:

  1. Check the pending orders 2-3 times per day, once in the morning, then around lunch and once in the evening (this depends upon the velocity of your orders and the amount of pending orders you’re seeing)
  2. You can copy/paste the pending orders list to excel (it doesn’t paste that well, but its workable)
  3. Take a few minutes to highlight new pending orders that have appeared and allocate stock levels accordingly.
  4. Track orders exiting to real orders and orders that are removed

A 5th option which may or may not be an option for your business (and I suggest you do so, because of the complicated risk factors with Amazon as we need to allow as much lea-way for returns and cancelled orders post Christmas) is that you actually order the products in the pending orders and allocate them to one side.

Summary

Amazon pending orders can be tricky, especially if your business is not using the MWS API or a 3rd party that uses this to communicate with Amazon & have that part dealt with effectively.

Stock can be allocated without the ability to let other selling channels know and this opens your business to a risk of overselling and the “fun” that stems from cancelled orders.

If you’ve not read this article yet (there is a video too), I cover some core essentials on how Amazon reacts to marketplace sellers. You never cross Amazon because if it was a human, it would be a woman and she is a ………

If there is only one takeaway from this article, go check your Amazon orders right now for pending ones, be aware of them and you can do this via this link.

Why I See eCommerce Differently. I see Data & Lots of It Too!

/by

Introduction

When I look at a website, an eBay item, an Amazon item or pretty much anywhere now, what “I see” when looking at such products is starkly different to most. In this article, I share with you what I see and also why you should be thinking along the same lines also.

I would class myself as one of the lucky few that see the world of eCommerce differently. I see beyond the pair of boots, shoes, phone or dress and see a multi-tiered inventory structure that sits behind it. I also see beyond a single businesses perspective and have a unique view of many and can combine this into a single view, that’s not distorted by a single viewpoint.

In this article I share with you what I see, to help you understand the data that is sat behind the inventory that you interact with daily and why I strongly feel and know that data is the lifeblood of an eCommerce business.

What I See When I look at an eBay Item

eBay Item Data Points BreakdownWorking from the top down, I have highlighted the data points that I see in an eBay listing in the image to the right (to help the full version is here) and I’ve broken these down in the bullet points below:

  • eBay Categorisation
    But not just that, two eBay categories, the category numbers and the listing requirements for the categories being used.
  • The main image
    This isn’t just ‘an’ image, its an image that meets the requirements to cause the highlighted bar beneath it for zoom and enlargement which can only occur when the primary image is over certain dimensions. Also, the main image has been worked on to show the variations in the listing and also watermarked too.
  • Additional Images
    The images in this example have also been sent to eBay and using their hosting services, plus they have been worked on externally to get them to an excellent level (eg backgrounds removed)
  • Listing Title
    A keyword rich title that has been created to maximise the exposure from users searching. It’s also worth noting another data point here would have been the subtitle, but this listing is missing it.
  • Item Condition
    The condition of products has become mandatory in numerous categories now and there are several options for this.
  • Variations
    These require a special mention and the data structure sat behind this cannot be comprehensively covered in this article (and I am working on an article specifically dedicated to this alone). However for now, understand that what we are looking at is a master/container inventory record that has the core details about the inventory line, however its variations (and associated stock levels & pricing) are shown in both the “Exact Colour” and “Size” variation boxes. Some categories require for these variations to be labelled precisely, however other categorise the inventory data can be more flexible.
  • Quantity
    As this is a listing with variations, each variant has its own stock level, for example a small sized item may only have 2 in stock, whereas medium and large of a specific colour will most likely have different stock levels as sized/colour/variations all tend to sell in differing velocities.
  • Price
    This listings price is straight forwards, there is only one price for the entire style of products being shown. However, each variant may have a different price associated with it and attached to the data structure sat behind it.
  • Postage Variations
    I don’t see just £2.99, I see a tiered pricing structure that is likely to have come either a business rule for a templating system of some kind, that is displaying the default shipping method (as an eBay internal code) and a value. Plus the potential for 2 more domestic options to be selected, up to 3 more options for international and additional rules or specifics attached to them (eg a rate that is set specifically for Ireland). I also see either the postcode requirement or sellers location, along with the despatching country from a table of options, so that such an ETA of delivery can be made effectively.
  • Delivery ETA
    With each listing a data field needs to be set so that eBay are able to show the estimated delivery time based upon the sum of the sellers handling time and the default handling/delivery time of the shipping service selected as the primary shipping option by the merchant. This is why we see 2-3 working days as an ETA, not 3-4 if the merchant had included an extra day for their internal handling.
  • Payments
    Again I don’t only see the option that has been set to “true” to accept PayPal, I also see two more options, the first is that the merchant has added in the additional payment option of postal order/bankers draft, but also there is an account wide setting in their account that is set to “true” and thus displays PayPal as their preferred payment method.
  • Revisions
    This listing is handy as it shows that the merchant has also included revisions to the listing. This could have been manual edits or the entire reposting of the listing data itself from a 3rd party application and eBay have kept a record of this.
  • Item Specifics
    The item specifics being shown in this eBay listing are relatively light compared to what could have been filled out. Looking at these I see additional fields that can have either fixed options to variable options completed for them. Taking brand for example, eBay suggest a list of common brands to use, however in this case the merchant has selected to use “Unbranded” as their data point.
  • Listing Template
    This is hard to explain if you’ve had no interactions with templating based system. Looking at the eBay listing below the item specifics line where the bulk of the inventory data is displayed, the logo and all the stuff that makes it pretty, I see a HTML template that is most likely laced with keywords that are populated when the listing is sent to eBay from multiple data points.For example the description is pulled from data field, the smaller item specifics table from multiple data points, the image, even the description from another source and the about us/payment/shipping/returns tabs of data all being brought in from multiple locations, combined together and posted to eBay as the final finished product we’re now looking at.The same goes for the eBay categories down the left, I don’t see categories, I see a table of category names and eBay shop category ID’s that have just been spilled out into a human usable interface. Depending upon the 3rd party application that is being used to create the actual listing, I also see post listing logic. For example in more complex listings there may be IF/NOT logic that defines what is and what is not being displayed. Such functionality can completely change the entire way the data is displayed in the template and the final listing we’re looking at.
  • 3rd Party Attributes
    This listing has two, the first from the team at DZine-Hub and the second from eBay’s Selling Manager Pro.  Its not uncommon for several 3rd parties to be attributed at this part of the listing.
  • Business Seller Information
    While not posted with the entire eBay listing. I see this data coming from the settings that the merchant has specified in their eBay account. Again this is just data fields that have made it to the final version of the listing we’re looking at.
  • Returns Policy
    Having a different view from is also highlighted here, this data could have been sourced from an eBay default in Selling Manager Pro or its come from a 3rd party tool that has stored this data also as a default, most likely with the other template data.

You should now have an idea on how I see eBay listings and I can assure you its not in the same manner you have been thinking about it. I glaze over the product actually being sold and look at the use (or abuse) of the data that actually made the presented item(s) we’re looking at right now.

Ready for the next one? Lets continue…

What I See When I look at an Amazon Item

Amazon is special, well different when directly compared to eBay inventory data. Its different because the inventory data could have come from multiple sources, Amazon directly, that of merchants, a combination of both Amazon and multiple merchants and also user data as well.

To break up an Amazon listing into a more clearly defined image for you to understand, I have used two colours for this image and again the full version can be found here.

  • Red is merchant or Amazon product data
  • Blue is user generated data

Amazon Inventory Data BreakdownMerchant / Amazon Inventory Data

For this item specifically, because its already been created, then it could be looked at much more simply. What I mean by this is that due to the nature of how Amazon’s product database works, the merchant doesn’t need to know all the attributes about the inventory record, just the essentials to enable it to be sold.

These essentials are:

  • ASIN
  • Merchants SKU
  • Inventory Count
  • Selling Price

This makes selling on Amazon for merchants exceptionally easy. If they can match their data to an existing inventory record then listing a single or many thousands of items is very straight forwards.

However for completeness for this article, I’m going to assume that this item has been created by a single merchant so that we can explore the data sat behind what we’re looking at in full.

Merchant / Amazon Inventory Data

This section relates to all the red highlighted content in the image to the right.

Running with the idea that we (as a merchant) need to create this record from the beginning, when I see this listing, I see the following:

  • Root Categorisation
    Amazon works slightly differently when it comes to inventory data creation. Each root category (Apparel in this instance) has its own data requirements and then below that something called “Browse Nodes” which can be compared to eBay’s category structure.By the very nature of this item (being a shoe) its in Apparel category and is actually a sub version just for Shoes. The “Browse Nodes” don’t actually appear in this listing, but they’re there stored in the background.
  • Product Title & Search Keywords
    The product title is just one factor in the items being found in Amazon’s search, underneath this are up to 5 search keywords that essentially extend the title further. So just looking at the products title, I not only see a listing title, I see 5 data fields that are sat with this, they’re just not shown in the front interface.
  • Pricing
    I’ve purposefully chosen this record because there is a variation in not only the products (size and colour) but also because there is a variation in pricing from the numerous merchants that are offering this product for sale on Amazon.Unlike the eBay example where it was just a single merchants product being shown, we are looking at the cumulative of multiple data sets and multiple merchants all hijacking the same record to sell the variants of this product. As such we can see there is a price range from £23.50 to £86.84.While on the topic of pricing, I see much further. I see a minimum price and I see a maximum price, I also see that there is likely to be repricing software running in the background that also needs these and maybe extra data points to move the pricing around to achieve a higher, but profitable number of sales for each merchant. I also see such 3rd party applications with large data sets and complex rules, all in the aim of profitability for such merchants.
  • Variations
    With regards to Amazon there are four variation types, none, size, colour and size & colour. We’re looking at the latter and we have both size and colour variants being shown.However it doesn’t just stop are text fields from defined lists,  if you look at the colour options, you’ll see that some of these have smaller images, these are called swatches. When I see these I see additional data that has been processed by one or more merchants to improve the visibility for the user when making a selection.
  • Main Image(s)
    Again assuming that the merchant is to be posting this entire product and its variants from scratch, I see numerous images all held in data cells, but not just images of black shoes, I see rows and rows of data, each with their own sets of images and hopefully swatch images (for the variant) with them as well.
  • Attributes /Product Specifications
    Different Amazon root categories (not browse nodes) have their own requirements and available options for specification extra data that appears in both the product specifications section, but over-spills to the product details  as well, weight is a good example of this along with the item model number.
  • Bullet Points (missing from this example)
    Missing from this item are the display of upto 10 bullet points that describe this product further. Most merchants include just 5, however its possible to include many more than this via the API. Again all I see is cells of data that are being posted by a merchant and in this case not being displayed back to the user interface.
  • Product Description
    Amazon by far as the most primitive (or restricted depends on how you look at it) method of accepting descriptions. Again this is just a chunk of data that may just be a wedge of plain text or it could be the combination of multiple data fields being joined together to make a description. This product is quite boring in this respect as there are some fabulous examples of merchants that I have worked with to bring in numerous data points to the description to make the product more effective.

User Generated Data

This section relates to all the blue highlighted content in the image above.

Amazon is again special when it comes to product data because as shown by the sheer amount of blue in the image I’ve included there is an awful lot of user generated data that while not pure to the merchant side of inventory data, is when it comes to when the product data is displayed back to the user (especially after time, as more and more user data gets added to the original inventory data).

  • Reviews & Likes
    I’ve joined these together, although they are separate parts. Likes is just a counter of users hitting a like button, however reviews… they’re special. This is user generated content that can be absolutely lethal in the conversion of products shown on the marketplace. Personally I have bought expensive goods on the reviews of people I have not and will not ever meet. However if they’re bad then that’s pretty much sealed the fate of the product.
  • Customer Images & Videos
    The example above is actually a semi-decent one, as beneath the main image there is a link to 6 user generated images, these are images that users/customers have uploaded and have been added to this product. You can also add video content to products too.
  • Tags / Lists
    Lists aren’t really shown in the example, however an additional component to an Amazon inventory record is in the inclusion of customer generated wish lists, however tags are shown and these are again generated by users to help others locate similar products.
  • Purchase/View Driven Data
    Now this is where Amazon excels. Using the viewing and sales data (probably other sources too), they manipulate this data back to add several key features to a products detail page, the most obvious is the “Customers Who Bought This Item Also Bought”, however bundles can also be automatically created and shown to the user (not shown in this example), along with numerous other sections such as “What Other Items Do Customers Buy After Viewing This Item?” and other related products.
    The data that is driving this may not entirely driven by the actual product data a merchant(s) first created the record with, but multiple other data points to make the final detail page more productive to the conversion of the viewer.

What You Should See

It doesn’t matter if I’m browsing for a book on Amazon or in a “real” retail store, I can’t help it, I just see data. That can of beans, its a SKU and it has attributes (shelf quantity, store quantity, warehouse quantity, distributor quantities, variation, container size, price, cost, margin to name but a few) . That book I just downloaded to the kindle, its a SKU and it has attributes. The parcels that arrived this morning, just a by-product of inventory data.

This is only inventory related and only the tip of the iceberg. When you mix in multiple data sources, the complexities of data that needs to be ported to not just one marketplace, but many and then the order related data that then is obtained by making inventory data visible, business processes, supply & distribution chains let alone the countless other messes of data such activities creates. It gets deep and fast.

The way I look at the world of eCommerce and product driven commerce, looks completely different to most and I hope I have given you a glimpse of what I see when I see a lonely product shown on a website or just a tin of beans on a shelf. What do you see?

Installing Firefox & The iMacros Addon + Speed Tip’s

/by

The video below takes you through downloading Firefox and the iMacro’s extension, plus some speed tips to improve the performance of your macros in Firefox.

 

Installing Firefox & The iMacros Addon Video

 

 

Helpful Links

How To Get on eBay Daily Deals

/by

“How To Get on eBay Daily Deals” and its deviations appear to be a common question that are driving people here, but I’ve never actually answered the question directly, only indicated in an earlier article that you can create your own using the eBay shops tags in the article called How to Make Your Own eBay Daily Deals & Weekly Deals.

In this article, I cover the ground around daily deals and try and be constructive as possible around helping you try and obtain a daily deal on eBay.

What Are eBay Daily Deals?

Today OnlyeBay daily deals have been around for a number of years now, there was a slight pause in these just before Christmas 2010 and they’ve been on the site since their re-introduction.

They’re focused towards amazing deals on products that also have a depth and desirability towards eBay users, whether you like it or not, most top spots are taken by larger outlet sellers who can deal with large volumes of transactions and have the inventory depth & buying power to deliver such deals.

Typically there are only two deals per day and are subsidised by weekly deals, that last funnily enough for a week. This is an important note to make, when we get to the pivotal question later, there are daily and weekly deals. One of these formats can be more productive than the other and it’s not the one you immediately suspect.

When we look at some of the recent daily deals further on in this article, you’ll notice that some of the numbers are quite attractive, to give you a wider picture, the volume of sales created by eBay are often dwarfed by that of sites that are wholly dedicated to daily deals, such as Groupon and LivingSocial.

Over 170 SoldCurious question, as what’s been featured in an eBay daily deal has been varied to say the least. From bikes to shorts, wellies to TV’s, XBOX’s to perfumes, it’s all been on a daily deal.

It’s not very well known, but there is an RSS feed of the eBay daily deals that can be found here: http://deals.ebay.co.uk/DailyDeal.rss this feed is very handy, especially if you keep track of it in a RSS reader such as Google Reader, as it’ll keep a history all the deals, so you can go back over them and see what’s sold and what’s not.

Note: Its not unknown that several companies keep track of the daily deals on a daily basis to leverage the sales information that they find through this channel and for other reasons.

Looking at the last couple of daily deals, there have been some real flops and some winners. I’ve put these into a table to make displaying the data a little easier on the eye.

Recent eBay Daily Deals

eBay Title Value # Sold Total
DGM ETV-2493WHC 24″ Ultra Thin LED TV DVD FULL HD 1080p USB PVR Freeview  £139.99  461 £64,535.39
E4 MENS STONE ROCK AND REVIVAL DESIGNER DROP CROTCH CHINOS ALL SIZES UK RRP £55
(2 Listings)		  £17.99 288 +
207		 £8,905.05
Women’s Nike Revolutionary Support Sports Bra 226015 ***3 Colors Available***  £7.99 3312 £26,462.88
Smith & Jones Marques Hoody
(6 listings)		  £14.99 624 + 427 + 370 + 284 + 242 + 232 £32,663.21
Kylie Minogue Sweet Darling Eau de Toilette Spray 15ml  £6.99  95 £664.05
South Double Breasted Duffle Coat In Mustard  £9.99  184 £1,838.16
BUSBI BOLT 32GB USB 3.0 SILVER MEMORY STICK FLASH PEN KEY DRIVE HIGH SPEED NEW  £24.99  18 £449.82
Xbox 360 250GB Console – Matte Black Finish  £179.99  418 £75,235.82
Fujitsu AH530 15.6″ Intel Pentium Win7 Laptop  £299.96  970 £290,961.2
Intempo Retro DAB/FM Radio with iPod/iPhone Dock in  £44.99  91 £4,229.06
Christmas Bumper Collection Book Pack – x6 Childrens Christmas Books  £8.99  44 £395.56
G-Star Raw Laundry Shooting Jacket  £39.99 143 £5,718.57
French Vintage Rustic Kitchen Set – Mug Mugs Chopping Board Magnets Shabby Chic  £9.99  48 £479.52
HIGH QUALITY EASY STORAGE BAMBOO CHOPPING BOARD WITH INTEGRATED PESTLE & MORTAR  £11.99  34 £407.66
HALO COMBAT EVOLVED ANNIVERSARY XBOX 360 MICROSOFT VIDEO GAME NEW SEALED PAL  £25.85  225 £5,816.25

As you can see its been a real mixed bag, the chopping boards were a complete flop, along with a few others, however in transactional volume, the laptops came out on tops with close to £300,000 of sales in a day or so.

This is the general theme of eBay daily deals, they tend to either flop or do “OK”, however there are always a few exceptions and when they do go, they will go deep in either the value or number of transactions (like the laptop’s v’s the bra’s).

The Issues With Daily Deals (as a Merchant)

The transactional volume of sales is an issue, as the volume of orders that a daily deal has been proven to produce and with the deal sporting an uncertain outcome.

Take the bra’s for example, that’s over 3,000 pairs to get out, ideally in under a day and that kind of spike can wreak havoc in a business if they’re not prepared (and even if they are!).

Daily deals don’t happen overnight, they’re scheduled with plenty of time lag to prepare for them and of course there is a negotiation period between the merchant and eBay. Even still the companies that I’ve worked with that have had both daily and weekly deals, have in some cases still struggled with the volume of orders that such offers can produce.

Preparation is key, the obvious one is packing as many lines as products as possible before hand, the second not so obvious is having a tool that can cope with a potential large volume of orders and of course the custom service requirement, as you know full well, this is eBay and for every sale, you’re likely to get at least one question ;-)

How Can You Get an eBay Daily Deal?

So we’re finally at the pivotal question “How Can You Get an eBay Daily Deal?” and sadly the answer is not straightforward.

Looking from side of eBay you’re going to have to check a few boxes, these could be:

  • Proven sales history on eBay
    You’ve got a long history of selling on eBay and that you’re capable of dealing with the style of buyers found on eBay (demanding).
  • Ideally at Strategic or Enterprise account status
    eBay Outlets tend to fall into the “Enterprise” bracket of account management, however there is a layer below this level and you’d be surprised at the number of eBay businesses that are covered by these account management levels. Instead of one account manager to several hundred accounts, we’re talking one account manager to less than 10 accounts. Now that’s some focus ;-)
  • Be sat on or have access to a product or range of products that can be offered an unbelievable value
    Its not unheard of for multiple offers for daily or weekly deals to be rejected, this is actually the hardest step to get through as the eBay daily deals staff are exceptionally picky in what makes it through to the final deals that are shown on the site.
  • Have the ability to process a very large spike in sales volume logistically
    Having back-end software tools that can cope with “volume” are almost absolutely required. While some merchants have used SMP (Selling Manager Pro) successfully, dealing with such volumes with this kind of tool isn’t really going to stand up to the job. Instead, if I was sat in the place of the person(s) that are calling which eBay daily deals to let through and which ones to not, I’d probably choose a slightly more uncompetitive one where the merchant has software that can cope with huge volumes and staff & processes in place to deal with them, over another merchant that I’d have doubts in.

Also an interesting dynamic that you may have not considered is that eBay account managers are likely to have sales targets to hit, this is an entry point you may not have considered as with just a few or even just one daily deal “in the bag” for them, could easily allow them to reach those targets and thus provide a motivation point for you to explore.

Now before you even consider hitting the ‘contact Matt’ page and asking whom to approach, the response you’re likely to get back isn’t likely to be particularly constructive. Instead, think about what internal channels you can leverage, such as the eBay account manager route and also what external channels such as LinkedIn to locate the people you need to contact.

Summary

In this article, I’ve covered what eBay daily deals are, what kind of volumes in both sales and item counts you can expect and perspectives you’ve probably not considered and this article should at least make you think of a couple of action points:

  • How can your business deal with +3000 in a single day
  • How can your business create a deal that is of exceptional value to eBay customers
  • How can you make your business & offers attractive to eBay

Now with that said, you should ignore daily deals, the deals you want to go for are eBay’s Weekly Deals and I explain these in one of my next articles.

Ted.com Massive-scale online collaboration

/by

I rarely watch “normal” television now and when I do it tends to be more factual rather than soap (as they drive me up the wall). Ted.com has been a fabulous resource of viewing material.

As such I would like to share a recent video from the site, that took and 10 second task and made it epically useful. I’m not going to say any more as it’ll spoil the presentation:


Data is the Life Blood of an eCommerce Business

/by

Introduction

Firstly let’s quickly identify whom this article aimed at and why on earth you should even consider reading the rest of this article.

I’ve personally spent over four years at 3rd party software providers working with businesses that have got themselves into operational & data issues that could have been avoided when it comes to marketplaces such as eBay & Amazon.

Some businesses have had no tool prior to moving into them and some have done but have made fatal mistakes along the way. Such mistakes have cost them untold amounts of wasted resources. This article is is aimed at helping you identify such issues as early on in the process as possible and to help you understand that data is the lifeblood of an eCommerce business.

New Businesses / Proof of Concept
This article is aimed at a brand new business that has just started exploiting the marketplaces for the first time, after all you need to have proof that the channels work and you need to start from somewhere.

Manual processes at this stage are a “good thing”, there is so much to be learned and as such the ‘manual’ entry experience is absolutely required, so that the people in the business understand the requirement for a tool as soon as possible and that they’re able to scale their operation.

Take the points on board now regarding data and look to the future of your business.

Businesses that have proven the marketplaces work for them.
This article is also aimed at businesses that have already proved that the marketplace channels are viable for them and are paying attention to them each day to manually load on products to the channels and most importantly processing orders from them too.

There is one exception to this, this is a business that has carried out substantial intelligence on the marketplaces and needs to leapfrog the basics and move straight to a large scale model as quickly as possible. If this is the case for your business, then you need to know how valuable data is and why you should retain it when it comes to the popular marketplaces.

As Your Business Grows

As the business grows bigger and bigger, normally by working (harder) on achieving one or all of these factors:

  • More inventory
  • Better priced inventory
  • More staff within the team
  • Adding more sales channels
  • Higher quality descriptions
  • Better order management
  • Lower operational costs
  • More profit
  • etc…

The cracks within the starting phase start to show very quickly, some identify this early on, some never identify it at all. Some unknowingly start to tackle this issue, but don’t actually realise why.

These could be through micro-management of staff & processes, after all most small businesses are rarely started & run by highly experienced managers, but by normal people and haven’t had the training or experience in dealing with such issues and I know personally that when I ran my own business, this is a trap that I got myself stuck in (and one that I’ve done my darn hardest to help others realise).

Putting internal management part aside and the other points above, what a small business owner(s) rarely tend to recognise is that “data” is one of core the reasons why they’re able to operate a business at all.

Data in a Business is Like Blood in a Human

Without blood, we’re pretty much screwed. The same as data within a business. Data is the lifeblood of the business.

This also ports to non-eCommerce businesses too, however is amplified by the very nature of the “e” part in eCommerce.

This article is entirely aimed at helping you understand that when you’re processing your orders in say eBay’s Selling Manager Pro, loading inventory to Amazon via spreadsheets or Seller Desktop, you are dealing with data. Your businesses data.

So when you’re ticking the boxes to print the orders out in SMP, using the sell your item form on eBay or adding items manually into Amazon using the add a product link. You are working with data, data that you should have absolute control over. It is after all the blood of the business.

Also the data that you are creating is exceptionally valuable to the business, the most obvious data is inventory data, however the not so obvious is the order data, that provides a unique history and insight to what the business has done.

So the moment you manually add inventory data into a marketplace like eBay or Amazon or process one or more orders in their web interfaces, you are potentially giving away the largest asset of the business to a process that you do not fully control or own and in the case of Amazon, you cannot get that data back out easily either.

An Example of Bad Data – Amazon

Let’s take Amazon as an example. It’s not unheard of for a business to have spent hundreds of hours loading thousand upon thousand of inventory records onto Amazon manually. And of course, this most likely had some huge positive effect to the business, namely orders & profit.

However what has been happening is that the business has been building Amazon a superb product database, that the business no longer has ready access to. You’re unable (either easily and certainly not officially) to export the product data that was originally created. While at the same time, Amazon are using that data to market not only your businesses data more effectively to more customers, but to other businesses too.

If you are manually creating products on Amazon using the web interface, stop immediately.

(More on this later in this article)

An Example of Bad Data – eBay

eBay UK LogoThe data requirements of eBay are huge. Lets taking the clothing categories for example. In recent updates the pressure to include attribute data as part of the listing process has become mandatory (along with many other categories too).

This compared to Amazon has positive effects, eBay are then able to leverage this data and then allow customers to drill down on their searches and ultimately makes a better buying experience and ultimately more sales for the business.

However, those attributes are now locked into eBay’s platform and getting them out again is painful and will cause a massive mop-up job of cleaning the junk out to make it “portable” later.

If you are manually creating products on eBay using the web interface, stop immediately.

(More on this later in this article)

Ironically in both examples it’s your data. Once it’s been entered into the said platforms, getting it out again in a format that is reusable is in some cases just not possible.

I’ve only ever worked with one company that elected to completely ditch all their product data to start again afresh. What they did make of it a second time around with a structured approach was truly amazing, but out of the hundreds of businesses I’ve worked with, this was the exception, rather than the rule.

What is more common is for businesses to try and do is to extract their product data and bring it into a 3rd party tool. In doing so (as I mentioned above) causes an absolute mess to try and sort out. The descriptions that come back from eBay are the full descriptions, so the data held within them is difficult to impossible to extract. Item specifics can be lost completely and any attribute data that formed variations can also be lost too.

In a recent conversation, it was mentioned that a business had tried using an import tool from a 3rd party software provider and the mess that was left over, not only went wrong multiple times, but was in such a state they were forced into abandoning it completely.

This is exactly the situation that you should avoid.

Step in the Saviours

It does not matter what software tool you use and this could of course be a combination of tools. The critical factor is that you have access to both the inventory data and sales data externally, outside of the marketplaces themselves.

To spell it out very clearly:

The moment you suspect the business is going to work long term, employ 3rd party software that is not created by the marketplaces as soon as possible or if it is, you can easily gain access to the original data.

Frankly I do not care what software it is and neither should you (to a point, that point normally being cost), as long as it allows you to enter the product data outside of the marketplaces and retain this data.

As far as sales orders go, having the ability to process them externally is normally a huge positive to the business, however in the case of Amazon, you are able export the sales data for current or later use, where as eBay, no real export and sales data can be lost after 45 days or so.

There are providers that can cost several thousand pounds over a year are not suited towards smaller businesses, however there are plenty of options out there and again the core criteria outside of cost, is:

  • Access to the raw product data
  • Access to sales data

Other requirements such as stock control, order management in the case of this article are either assumed or luxuries.

Free Examples

We all like free and here are two free options for both eBay & Amazon.

Amazon
If we take the Amazon marketplace as an example here, if you stop entering data manually and start using the import sheets they provide for the inventory data, because you’re storing this on one of the computers you have access to, after sending it to Amazon to create the product, keep the sheets, because you can use it at a later date, either to update the records that were in the file or to transfer the data to another marketplace/sales channel.

eBay
If you’re looking for a free option for eBay, then for product data eBay’s File Exchange this covers both product creation, updates and also order data too. However can be clunky and in massive accounts, can take several hours to process requests (I tried this on an account with over 10,000 live listings, it took 5 hours even for a basic export, on a smaller scale it would be adequate though).

Other Providers
There are a multitude of providers that can offer superior options outside of this, they are not all SaaS models (where you pay them a commission on sales) and if your requirements are low, then community extensions to popular website open-source products can be free or very inexpensive (covering these is not a concern for this article, just that they exist).

Ultimate Goal For Your Data

However the ultimate goal in regards to your sales and inventory data is that you have access to them and you can then re-use the data. I call this “portable data”, data that you can use as you need to, where ever that may be.

Sticking with the example approach of this article:

Lets say you start a business on eBay or Amazon, you see the signs that its going to go well and do put some form of software in place to maintain control and ownership of your data, you can then grow and grow as the business dictates. You can move software providers relatively easily and if you want to add in new sales channels, such as a website or another marketplace, with a few tweaks to the data you already have access to, you can do. The other option of not doing this, is frankly nasty.

Summary

Data either product or sales related is the life blood of an eCommerce business.

It needs to be “clean“, it needs to be “yours” and it needs to be “portable“.

If you keep bashing data into marketplaces and don’t retain a copy, then you’re basically capping the potential of you and your business. And that’s something neither of us desire.

Dealing With “Odd” Data Delimiters In Microsoft Excel

/by

In this video I cover the following topics:

  1. What are CSV files
  2. What data inside a CSV files look like
  3. Why the standard comma separator can sometimes not work
  4. An example of a different separator
  5. How to open such files in Microsoft Excel (at 4:00 minutes in)

Website Data Extraction/Scraping & Form Filling Expert

/by
Due to demand, this service is only available to existing clients and is no longer available.

Website Data ScrapingWeb scraping can be hugely advantageous to businesses, allowing them to function more effectively and keep up-to-date with information that is on specific websites more frequently and accurately.

This is especially true when you consider the applications that can be created can be run by numerous members of staff on an ad-hoc basis or even automated everyday at certain times or that they allow the access to complex data from suppliers for more effective merchandising or keeping their internal systems updated more frequently with stock and pricing information.

This can be also a very quick process too, taking only a few hours to complete most projects and then only taking just seconds for small projects to run and depending on the complexity & speed of the users connection to the internet.

I have several years experience with web scraping over many projects and requirements. In this article I cover the details of extraction in more depth and include examples where suitable. If you have a project in-mind, contact me today with your requirements.

I specialise on small & medium scale scraping projects, such as extracting data from supplier websites for product information, stock & prices updates.

However I can readily tackle multi-tiered extractions and also create clean data from complex situations to import into 3rd party applications with little to no input from the user.

Not only can most data be extracted from most websites, the data can also be posted to websites from data files such as CSV.  This could be form filling for job applications, listing products on to websites or online dating requests, not just extracting product, service or article data from a website.

If you can do it in a web browser, then it can most likely it can be automated.

The possibilities are almost endless.

If have a project in mind, Contact Matthew today, it could be completed in just a few hours.

 

Getting The Edge With Data Extraction

Using automated tools to grab or post data to the web could trim hours off each day or week. Extracting the latest stock & prices from suppliers could mean higher profitability and less back-orders. It could even mean reams of data from suppliers websites and give your business the edge over your competitors.

It doesn’t matter if its behind password protected content, if you can “see it” in your web browser, chances are it can be extracted. If you’re entering data manually into website forms, chances are high that it can be automated too.

I’ve worked on numerous projects where clients have been able to ensure that they’re back-office tools are up to date as possible with the latest information from suppliers and even allowed businesses to work with suppliers they’ve never been able to do with before, because the requirements to extract data from supplier websites has been too restrictive either due to time or cost.

Knowing what your competitors prices are can be a huge advantage when it comes to pricing especially in the eCommerce environment we have today. If you’ve got the data and they can be matched to other sites, within one click and a few minutes, the latest pricing information from competitors could be yours. As many times as you want, whenever you want.

Scraping & data extraction can solve this in a cost effective manner. One script, used over and over. Anytime you want by however members of staff you have.

If you want the edge, Contact Matthew today.

 

The Required Tools Are Free

Using two free applications, the first is the Firefox web browser and a free add-on called iMacros, simple to very complex web automation can be completed.

This allows completed projects to be run by the owner using free-to-use tools, so that any extraction or processing can be run by the owner or staff members as many times as they require and however often they require.

Also extract processing can be obtained using JavaScript to process complex data inputs or extracted data from websites. I cover this in more detail in the “extra data processing” section.

Don’t worry if you’ve never used either of these before, if you’ve used a web browser and can press a button, its that simple. I’ll help you get started and its very easy to do. I also include instructional video’s to get you set up. It’ll take no more than 10 minutes.

Simple Extraction

In this scenario, data elements from a single page can be extracted and then saved to a CSV file.

Example:

This could be a product detail page of a TV and the required elements, such as:

  • Product title
  • Price(s)
  • Stock number
  • Model number
  • Images
  • Product specifics
  • Descriptions
  • Reviews

Are all extracted and then are then saved to a CSV file for your own use.

The time it takes to make a simple extraction of data from a single page varies greatly, this is because the data on the page can sometimes be very poorly formatted or if there are lots of fields that need to be extracted this can take quite some time.

If have a project in mind, Contact Matthew today, it could be completed in just a few hours.

Extra Data Processing

Extra processing can be applied to the extracted data before saving to a CSV file. This is very handy when you only want or require cleaned data to be saved. Most of the time its obvious that cleaning is needed and basic cleaning of the data is included in the macro.

The quickest way of identifying any processing you require on extracted data is to provide an example file of how you would like the final data to look like.

Example:

If one of the extracted fields was a complex data field such as and email address held with other data in JavaScript, such as this:

<script language=”javascript” type=”text/javascript”>var contactInfoFirstName = “Vivian”; var contactInfoLastName = “Smith”; var contactInfoCompanyName = ” REALTY LLC”; var contactInfoEmail = “[email protected]”; </script>

Instead of including the extra information in the export, the email address can be identified and only that data field is extracted. Or if all the data held in the JavaScript is required, this could be split into separate columns, such as:

First Name,         Last Name,        Company Name,              Email Address
Vivan,                   Smith,                   REALTY LLC,                        [email protected]

Also, if the data needs to be formatted for import into a 3rd party application, such as ChannelAdvisor, eSellerPro, Linnworks or a website application, this isn’t a problem either. I’m exceptionally competent with Microsoft Excel & VBA and can help you leverage the gained data and format it in a complete solution that requires the least amount of input from you or your staff.

Even if you have basic requirements or highly complex Contact Matthew today, your data extraction project could be completed in just a few hours and fully customised to your business requirements.

Paginated Extraction

This can vary from site to site, however complex extraction could involve navigating several product pages on a website such as search results, then navigating to each product that is in the search result and then processing a simple extraction or a complex extraction on the products detail page.

Example (property)  – Website: Homepath.com

In this example, not only is the requirement is to extract the data found for a specific property; it is also required for ALL the search results to be extracted.

This would involve extracting all the results and then navigating to each property page and extracting the data on the property detail pages.

The time taken to extract the data from such pages varies on both the number of property results to go through and the amount of data that is to be extracted from the property details page.

Example (products) – Website: Microdirect.co.uk

In this example similar to the properties, the requirement is to extract the data from each of the product pages, but to also to extract the product details pages data for all the pages in the search results.

The macro would navigate through each of the page results (say 10), identify each of the products, then one-by-one work its way through the products, saving the data to a file.

Need data from pages & pages of a website? Not a problem, Contact Matthew today, it could be completed in just a few hours.

Ultra Complex Extraction

These normally consist of a requirement of data to be processed from a CSV file, then external processing & scraping by the macro and then possibly depending upon the results, further processing or scraping is to be completed. Such projects are normally very complex and can take some time to complete.

Working with multiple tiered drop down boxes (options) fall into this category, as normally by their very nature can be complex to deal with. It’s also worth noting that is possible to work with multiple tiers of options, for example, when making one section, the results cause sub-options to appear. Sites that need image recognition technologies also fall into this category.

However it’s easier to explain an example rather go minute detail.

Example

For this example, you have a CSV file that has a number of terms that need to be searched for on a dating website, once these searches are made, the details are saved and then it is required to contact/email of the persons through another form.

The macro will make intelligent searches for these terms and the matching results (these are likely to be paginated) are saved to a separate file. Then for each result that was saved, the macro will then are then sent customised contact messages through another form found on the same or different website.

Do you feel your requirements are complicated or the website you’d like to extract from or post to isn’t simple? Contact Matthew today, I’ll be able to let you know exact times & can create the project for you at a fixed cost.

Saving Data & File Types

Extracted data is normally saved as CSV files. The data is separated by commas “,” and will open in Microsoft Excel or Open Office easily. For most applications using a comma will work perfectly.

However sometimes, the data that is extracted is complex (such as raw HTML) and using a comma as the separator causes issues with leakage when viewing in Microsoft Excel or Open Office, this is when using other characters such as the pipe “|” comes in very handy to separate the data fields (eg title and image).

The separator can be any single combination of characters you wish, some common examples are:

  • Comma “,”
  • Tab ”      “
  • Pipe “|”
  • Double pipe “||”
  • Semi-colon “;”
  • Double semi-colon “;;”

It will be quite clear from the onset which separator is required either from the data is being extracted or the projects requirements. If you have any special requirements, please discuss this beforehand.

XML or SQL insert statements can also be created if desired, however this can add several hours onto projects due to its complexities.

File types an issue? I can pre-process data files before-hand in other applications id needed.  Contact Matthew today, it could be completed in just a few hours.

Speed of Extraction/Form Filling

As a general rule, the projects I create run exceptionally fast, however there are two factors that will limit the speed of them:

  • The speed of the website being interacted with
  • The speed of your connection to the internet

You can also make project scripts run much faster by ensuring that the following options in your iMacro’s settings are turned exactly the same as those shown below.

You can find the options page shown below by clicking the “Edit” tab on the iMacro’s side bar, then pressing the button called “Options”.

Imacros Option Panel

Even if you above looks complicated, its not. Instructional video’s are included and I’ll make it exceptionally easy for you. Contact Matthew today, it could be completed in just a few hours.

Exceptions & “Un-Scrape-able” Pages

It is important that your processing requirements are discussed before hand with examples, so that I can confirm whether or not automated scraping will suit your requirements. In most cases it will do, but sometimes it’s just not possible.

In some cases, it is not possible to extract data from pages over & over due to:

  • A poor ‘make up’ of the page
  • Inconsistent page layouts
  • Page structures that vary enormously from one page to another
  • Use of flash or excessive use of AJAX
  • User code ‘capture’ boxes (like recapture)

When this happens, then the only consistent method of extracting data from such pages is by a human and scraping will unlikely be suitable for your requirements. This is rare, but does occur. If I identify this (and I will very quickly), I’ll let you know ASAP.

I am unwilling to work with data of questionable content. The below above are just common-sense really, I’ve added them for completeness.

  • Adult orientated material (porn is a no, services are a no, ‘products’ are ok)
  • Sites that are focused towards children
  • Identifiable records on people such as medical records (order related data is fine if they are yours).
  • Most government sites
  • In situations where I suspect the data will be grossly miss-used for fraudulent or illegal purposes.

Unsure on what your requirements are or just not sure if web scraping is the right way forwards for your business requirements. Contact Matthew now, I’ll be able to help you and turn it into plain English for you.

What Are the Limits of Extraction/Processing?

Most normal limitations are caused by very large or very deep page requirements of a project. That doesn’t mean they’re not possible, just that it could take some time to code for and also for to run each time by you.

The projects that I create suit smaller scale situations, such as one off extractions or extractions that need to be run by the owner over and over, such as on a daily basis to collect the latest product & pricing information from a supplier.

The real limitations come in to force when the requirements are for huge scale extraction, such as hundreds of thousands of records or exceptionally complex and exceptionally deep extractions. This is when using tools such as Pyhon , C++, Perl or other languages that allow spidering of websites would be more suitable.

This is not a speciality of mine, however due to my experience with scraping, I can assist you with project management of such projects with 3rd party contractors. Contact Matthew now if this is what you need.

Anonymity & Use of Proxies

If you need to keep the activities of such scripting hidden to remain anonymous, then this can be achieved on small scale projects using free proxies with no interaction from yourself.

In larger or more repetitive situations then either I can help you setup your browser to use free to use proxies (can be unreliable at times) or in most cases I’ve found leveraging inexpensive a services that are very easy to use and most importantly reliable.

If this is a concern for you, don’t worry I’ve done it all before. Contact Matthew now if this is a requirement for your project.

Do you provide ‘open’ code?

For ‘small’ or ‘simple’ macros, yes the code is open and you or your development team are able to edit as required.

However for some complex or ultra complex macro’s the code is obfuscated due to the extra functions that are normally included. This is non-negotiable as I have developed custom functions that allow me to uniquely deal with complex situations of data extraction & posting.

Is Web Scraping Legal?

The answer to this can be both yes and no depending upon the circumstances.

Legal Data Extraction
For example if you own the site and data that is being extracted, then you own the data and you’re using it for your own purposes. If you gain permission beforehand, for example from a supplier to extract data from their website, this is also legal.

I have worked on projects where an employee has left a company and there is no access to the back-ends/administration consoles of websites and the only way of obtaining the data held on the site is by scraping. I’ve done BuddyPress, WordPress, PHP-Nuke, phpBB, e107 & vBulletin sites previously to name just a few.

Also I have completed many projects where product data is extracted for use by a business to obtain up-to-date pricing and stock information from suppliers websites, along with extra product & categorisation data too.

Illegal Data Extraction
Because the macro’s are run on your or staff computers, scenarios outside of where the sites are owned or permission has been granted, fall into your discretion.

I cannot be held responsible for any legal actions that may proceed from your actions of running such scripts on 3rd party websites. As such I strongly recommended that you contact the 3rd party to seek consent and check any privacy or usage policies that they may have prior to extraction.

Contact Matthew

If you’ve got a clear idea on what you’d like done or just even if you’re just not sure if its even possible, Contact Matthew today and I’ll be able to tell you if it is possible, how long, how much and when the completed project will be with you.