Screaming Frog is the industry standard SEO crawler, but it can slow down your computer and monopolise your internet connection. By running it, instead, on Google’s Compute Cloud it can run significantly faster and you’re free to get on with other jobs whilst it runs. Moreover, if you can run one Google Compute Cloud Instance, you can run many.
In this part 1 of 2, I’ll explain how to setup Screaming Frog to run in the cloud and then, in part 2, I’ll explain how to take it to the next level and use it to crawl potentially hundreds of websites, simultaneously, faster than you’ve ever done before.
Before you start this part 1, you’ll need:
a Google account;
a card that Google can charge for use of their servers (you’ll be able to use them for free for a while, but have to enter card details immediately in any case); and
a Screaming Frog license. If you’re planning to simultaneously use Screaming Frog on your computer and the cloud you’ll need two licenses. Screaming Frog is inexpensive – support its continued development.
Create a new project, calling it whatever you’d like.
Click on ‘Compute Engine’ ‘VM instances’ and create a new instance. Here are the setting you’ll need:
Once entered, you’ll be back on the virtual machine page and your shiny new virtual machine will begin booting. Click on ‘SSH’ ‘Open in Browser Window’ and you’ll have a box like this appear:
enter ‘sudo add-apt-repository ppa:webupd8team/java’
enter ‘sudo apt-get update’
enter ‘sudo apt-get upgrade’
enter ‘sudo apt-get install xfce xfce-goodies autocutsel tightvncserver’
enter ‘sudo apt-get install libxss1 libappindicator1 libindicator7 fonts-liberation’
Choose ‘start Cloud SDK Shell’ and ‘Run gcloud init’
In command prompt, run ‘gcloud login’
Once the above is complete, enter ‘gcloud compute ssh [the name of your virtual machine] –project [the unique name of your project] –zone us-central1-f –ssh-flag “-L 5901:localhost:5901”
It will ask you to set a password and then the connection will be made.
Installing & Setting up VNCViewer
I like RealVNC’s VNC Viewer, though it doesn’t really matter which you use. You can download it here.
Once installed/opened, click on ‘File’ ‘New’ and enter ‘127.0.01:5901’ as the VNC Server – name it whatever you like.
Once saved, click on the server’s icon and type in the password you set when you typed ‘vncserver’ earlier. A window should then open connecting you to your virtual machine:
And there you have it – you’re all set to run your first crawl. One important thing to remember is that you’re paying (or using your free trial credits) whilst the computer is running (and only a tiny amount when it’s not) so, when you’re done with the VM, shut down the terminal window, VNC viewer and in the Google Cloud Compute Console, stop the machine.
You’re ready to crawl hundreds of thousands of URLs without having to worry about it slowing down your internet connection or computer. BUT I wouldn’t recommend stopping here. Ultimately the above is a lot of work if all you’re trying to do is stop colleagues complaining about you bogging down the connection. The real power of this doesn’t come from running a single one – it happens when you automate the whole thing, running multiple instances at the same time, giving you a huge speed boost that’ll allow you to crawl entire industries faster than you’ve previously crawled single sites. Luckily, that’s a lot easier than it sounds. So jump over to part 2 here to turn your basic setup into something that’ll change the way you work.
Verve Search Introduces: The LinkScore Tool
Want to understand the real value of the links you’re building? Here at Verve Search, for the past five years, we’ve been developing a proprietary metric to do just that. Up to now, we’ve kept it exclusively for our clients, but, in the interests of transparency, and for the benefit of the industry as a whole, in the back end of last year we took the decision to build a free to use, public version.
After a whole bunch of work, we’re delighted to say it’s live and ready for you to play with.
For those of you who just want to get their mitts on it, it’s right here.
For those who want to learn more about how it was developed, read on!
Why did we build the LinkScore tool?
A single metric might not always tell the full story!
We’d always felt that there was probably little point (from a rankings perspective) in having a link on an amazingly authoritative domain if it’s no-followed and in a language that neither you nor your customers speak. Yet, if you use a single metric to determine the authority of a link you may find that you’d be treating those links as if they were of equal value.
As such, rather than use a single metric, our tool blends more than 10 different on and off-site metrics, in order to assign a value to a link.
We needed an international metric… We found that many SEO tools that assign link metrics are primarily focussed on English-speaking audiences. So, whilst their metrics might work well in primarily English-speaking countries, that might not always be the case in countries where English is not the native language. Therefore, we built the LinkScore to provide scores that give equal value to equivalent authoritative sites in each country – meaning quality links in one country are assigned an appropriate value.
We wanted a tool which could evolve & keep pace with the industry! Each of the different variables added into the LinkScore were chosen based on our own testing and benchmarking. Where we’re using third party metrics we felt it was important that we weren’t tied to one particular database, and as a result we’ve been able to choose multiple best-in-class metrics that get us as close as possible to measuring the true ranking value of a link. Over the years the LinkScore tool has continually evolved alongside this fast-paced industry.
What does the LinkScore tool do? It allows you to measure a link’s ability to influence rankings. It also allows links to be compared with each other, and groups of links to be compared periodically. Please note, we built this as an SEO tool, and as such, the tool does not take into account the value a link might provide in terms of PR, branding or any other type of marketing.
When you run a link through the tool a score between 0-500 will be returned. This scale is not logarithmic, however some of the variables used to calculate the score are.
Semantically relevant, followed, in-content links in unique content on authoritative domains yield the highest scores. Example sites which would yield high scores include the BBC and the New York Times.
How are scores calculated?
We could tell you, but we’d have to kill you! Kidding 🙂
We keep the exact metrics, and how they are combined, a closely guarded secret. This is to stop people gaming the algorithm, because it is updated annually and because we think the accuracy of the final scores speaks more to the quality of the LinkScore than any particular one of its metrics.
What do I need to do to use the tool? You’ll need to input your Majestic, Dandelion and SEMRush API credentials and you’re good to go. Why? Well, the LinkScore uses metrics from each of these providers as part of its algorithm. To prevent abuse of the tool, we require users to use their own API accounts rather than providing free access to our own. Rest assured, your API credentials are stored locally on your computer; we do not keep a copy of your API credentials, nor do we use them for any purpose other than analysing the links you add to the LinkScore tool.
How much does the LinkScore tool cost? Except to the extent that it uses your third-party API credits, the LinkScore tool is free to use. Rate limiting may, however, be put in place to maintain the experience for all users.
Do we store your data? Definitely not! We do not store your API credentials, the links that you run through the LinkScore tool, or the score output. However, we do run Google Analytics and so store a number of different metrics related to your visit including, but not limited to your location, browser, time on site and pages visited.
Got more questions? Check out our FAQ, or contact us.
And if you do play with the tool, do let us know what you think.
outREACH Conference
One dreary morning in January we decided that it would be a good idea to host a conference in the summer, you know, just to challenge ourselves a little. The next day the venue was booked.
A week later we’d got the branding sorted and had put up a page on the website. Now there are just a few days to go until outREACH, our first ever conference!
We are so humbled by the positive response we’ve had from brands, other agencies and freelancers, and we would like to thank everyone who has bought a ticket to support us. We cannot wait to meet you all on Friday!
As seasoned speakers, Lisa and Hannah knew exactly what was missing from the conference scene – a single track event 100% dedicated to outreaching content. Our event follows the whole process, with sessions on coming up with creative campaigns, how to get your ideas signed off, mindset, how to (and how not to) approach journalists, processes, tools, tips, and so much more. There is no other event like it in the SEO conference calendar.
We’ve hand-picked our speakers and the entire event has been planned around their specialist knowledge. You can see a full list of speakers on our agenda page here.
Jim will show you how to tame your tigers
Closing the conference is our keynote speaker, Mr Jim Lawless, with his tales on how he used the right mindset to ‘tame his tigers’ and become a jockey…oh, and also get in the record books.
outREACH takes place this Friday, 9th June, at the Congress Centre, 28 Great Russell Street, London WC1B 3EN.
If you would like to purchase a ticket (we have less than 15 remaining), click here and enter the promo code LASTCHANCE and you will be able to secure your ticket with a 30% discount off the ticket price.
We would really love to see you there!
What Makes News?
“How do I get a journalist to cover my topic?”
This is a question that PRs constantly ask.
Experience teaches a lot about what news media’s preferences are, but learning what the definition of news is can be rather helpful as well:
The definition of news is simple. News can be defined as information about current events.
Not all information about current events is news though.
A piece of current information becomes news when it is new, unusual, interesting, significant and about people. This can be an event, action or occurrence.
Let’s look at a few examples to understand better what actually constitutes news:
‘A bus driver drove kids to school this morning’ is not news. This piece of current information is about people, but it’s nothing new, nor is it unusual, interesting or significant.
If the bus driver hadn’t turned up to work in the morning and the kids were left at the bus stop waiting for their lift to school, it would be news , this event is at least unusual, maybe even interesting on a local level.
The aforementioned example also proves the importance of newness. The fact that the bus driver did not turn up to work and the kids were left to make their own way to school is news on the day it happened, and possibly in print 24-hours later.
News has to be something new. Unusualness or significance doesn’t help if the piece of information is already known to the general public.
To make headlines outside of a local newsbeat, the bus driver story would need greater significance.
The significance could be increased, for example, by very poor weather conditions , if the kids had to wait in an ice storm, the event would have been more widely significant and, therefore, a lot more newsworthy.
The involvement of people is key in this example as well. If the driver didn’t turn up to work but there were no kids at the bus stop waiting, the bus driver’s absence would not make news.
Sometimes a story does not require people to be the subject , like a hurricane for example , but the hurricane only becomes news if it touches people, or something connected to people, such as a house or bridge. In an uninhabited area, a hurricane is not news, but as soon as it threatens a town or a city populated with people, it becomes newsworthy.
So how do we apply this in our work?
Last autumn we created Polar Night Counter for Expedia Finland. It is a tool that counts down the days to the polar night , a period of time when the sun doesn’t rise above the horizon , and tells the length of the day in different parts of Finland.
The annual light phenomenon inside the polar circles has hardly anything to do with people , and one can’t call it very unusual either since it happens every year. How did we then manage to get any coverage for it?
The polar night isn’t about people but it definitely has an impact on people.
Just imagine being in Utsjoki, the northernmost town of Finland where in winter the sun doesn’t rise in 52 days. The dark period unarguably affects people , some say it makes them tired or even depressed, whereas others cherish the beauty of the mysterious blue light it creates.
MTV.fi is the online news site of Finnish MTV, one of country’s biggest commercial TV channels.
We turned the piece into news by listing the duration of polar night in northern cities and towns in Finland. We thought this would allow the newspapers’ readers to see clearly what the polar night means to them , in other words we made it clear how this phenomenon has an impact on people.
By picking the most interesting and newsworthy pieces of information and presenting it in an easily readable format we made it easy for media to cover the topic. The timing was also key: we contacted media early before the start of the polar night , because the start of the polar night is only news when it happens, not a day or two after. Polar Night Counter wasn’t only covered by local publications, but national too on the eve of the start of polar night in Utsjoki.
Ilta-Sanomat is one of the biggest online newspapers in Finland
So much about news! Hope this example helped you to understand the concept of news better.
Verve Search wins Best Large SEO Agency in Europe!
It’s official, we’re the best Large SEO Agency in Europe!! (It’s not bragging if it’s true!.)
On Thursday 27th April, eight members of the Verve Search team (plus two of our lovely clients from Expedia and Gocompare) flew from London to Krakow to attend the 6th Annual European Search Awards 2017. As usual it was a glitzy and glamorous affair, with lots of polish vodka with some pretty stiff competition to match. We were honoured to be nominated a total of nine times.
What makes us the best SEO agency in Europe?
We launched 50 creative campaigns in the last 12 months, generating over 1,780 high authority links from the biggest authority sites in the world including; BBC, Wall Street Journal, The Guardian, New York Times and many more.
Additional offline coverage including; ITV 8 o’clock News, The Observer print coverage and even tweet by NASA.
Average of 40% increase in organic visibility YoY for our clients.
What makes an award winning agency is the people that work here and our fantastic clients. We genuinely feel like an extension of our clients teams and vice versa.
Our vision is to be “THE destination for digital pioneers”, we work hard to make that a reality. The talent, dedication, drive and most of all the mindset, is what sets us apart. We make shit happen!
Whilst part of team Verve were in Poland, the remaining ‘Ververs’ were at our local pub in Surbiton, following the action on Tweet Deck. We could feel the love and support from 1,021 miles away.
We are very PROUD to have received this prestigious award and dedicate it to the everyone in the Verve Team and our amazing clients. Thank YOU!
Lisa Myers recognised in the BIMA 100 list 2017
We’re excited to announce that our very own CEO, Lisa Myers, has been recognised within the ‘CEO and Leader’s’ category in the prestigious BIMA 100list.
The BIMA 100 recognises the best in UK digital and celebrates individuals who are shaping the future of Britain’s digital industry.
Here are some things we LOVE about Lisa:
She has great passion and believes that we are able to do anything if we put our minds to it (a VR campaign on a Norwegian Train? , Sure!)
She’s inspiring , she set up her own SEO Agency eight years ago and has grown the agency to become one of Europe’s leading SEO agencies. Winning Best SEO Agency two years running at the European Search Awards.
She never gives up. She’s like Yoda really.
She cares very much about people and culture; she strives to develop people in the very best way she can. She also has an uncanny ability to spot talent.
She does an awesome RUNNING MAN on occasion.
But seriously, she deserves this nomination and we are very proud to call her ‘the boss‘.
Congratulations to Lisa and the other people recognised in the BIMA 100, enjoy your night in London on the 9th May!
4 Awards + 10 Nominations for UK Search Awards 2016
It’s been a great week. Scrap that, it’s been a FANTASTIC WEEK!
On Tuesday night we WON 4 awards at the Digital Impact Awards:
Our campaign “Accent Map” for Expedia UKWON two Gold in the categories: Best use of SEO in the Corporate Sector AND Best use of Digital in Travel & Leisure.
Our campaign “Billionaires’ League” for GoCompare.com won Silver in the category Best use of SEO in the Corporate Sector and Bronze in the category Best use of Digital in Finance.
Want to understand the real value of the links you’re building?
We launched 50 creative 
The BIMA 100 recognises the best in UK digital and celebrates individuals who are shaping the future of Britain’s digital industry.
She has great passion and believes that we are able to do anything if we put our minds to it (a
