Screaming Frog – what is it?

Screaming Frog is nothing more than an advanced crawler that scans websites for a comprehensive analysis of their structures and elements affecting on-site SEO. The tool facilitates the work of auditors because it detects numerous irregularities in connection with meta tags, duplicate content, headers (H1-H6), 404 errors or even incorrectly implemented redirects.

Free version of Screaming Frog
The free version of Screaming Frog allows you to e.g. generating XML sitemaps, analyzing 500 random URLs from a selected website, as well as visualizing specific domains.

The paid version of Screaming Frog
The paid version of Screaming Frog offers much more and costs around $ 150 a year. At this price, we can, among others analyze any number of URLs, render JavaScript, integrate the tool with Google Analytics / Google Search Console, and even check spelling and grammar on the page.

Screaming Frog – how to use?

To start working with Screaming Frog, enter any URL that you want to be scanned by robots (Mode -> Spider / Mode -> List). Then, it is worth learning how to use the basic functionalities of this tool to facilitate the collection of information on the subject of the website, which are presented below.

Title tag and meta description, H1 / H2 headers
The tool provides us with information about duplicate or missing parameters, possibly not matching the Google guidelines. We can analyze this type of data by entering the target URL. In the SERP Snippet tab, we will check how individual tags are displayed in the search engine.

Verifying broken or redirected sites
This functionality is especially appreciated by Paraphrase-Online.com on-site specialists. Well, it allows you to find all broken pages within the site. That’s not all, because if we click a specific address – we will verify where the links to pages returning the 404 code came from.

Checking the loading time of domains
The third, very useful option of Screaming Frog is checking the loading time of domains. Just go to the Internal tab, then HTML, and then click “Response Time”. Then we will be able to sort the data based on the loading time and of course compare it with the GA (Google Analytics) report on the bounce rate.

Remember!
Pages that load much slower tend to have a higher bounce rate than sites that load faster.

Information on internal links that lead to specific pages
An important point for every SEO specialist specializing in link building. Well, thanks to the Screaming Frog tool, we will verify each URL address based on internal links leading to it, as well as text anchors. Just select the “Inlinks” tab at the bottom of the program to find out, among other things, what pages are the links coming from, are they dofollow / nofollow links and what anchor texts have been redirected to the site.

Additional “Tree” view facilitating the analysis of the website structure
ccording to Paraphrase-Online experts, the analysis of the website structure, using the additional “Tree” view, is very useful because it allows you to verify what the URL of each subpage looks like. Thanks to this, we will be able to eliminate incorrect navigation elements and implement the necessary structural changes.

Checking the correctness of the implementation of Google Analytics and Google Search Console
Screaming Frog also allows you to verify the implementation of GSC and GA tags on each of the subpages of the selected website. It is also worth knowing that the columns available in the program can be rotated and placed in the correct order. As we mentioned before, all information can be easily exported to Excel.

Create an XML Map
The functionalities of Screaming Frog allow you to instantly create a graphic sitemap and XML sitemap. If we want additional configuration – we will determine the frequency of visits to the site by the robot, which will be based on the place in the domain structure of a specific subpage.

Remember!
We can open the sitemap in Excel. This is very useful because we will verify that the specific addresses do not contain duplicate parameters. That’s not all, because for key URL links we can set the most important guidelines manually (e.g. frequency of changes, priority, and other values). After you finish working with the sitemap, save the file as XML.

It is also worth mentioning the possibility of verifying information in the context of website graphics (eg images exceeding 100 Kb, with a missing “alt” attribute or exceeding 100 characters). Screaming Frog SEO spider also allows you to check the correctness of URLs and list errors, e.g. URLs exceeding 115 characters, including underscores or uppercase letters.

How to speed up the analysis in Screaming Frog?

To speed up the analysis in Screaming Frog, you should divide your work into stages – then we will be able to initially verify selected information about a given website, and then use the additional functions of this tool. First of all, we recommend paying special attention to, among others:

– page size,
– the number of dofollow and nofollow links,
– number of internal and outbound links,
– page description (meta description),
– page title (meta title),
– keywords,
– status (there is 200 or 404), – page encoding (UTF),
– domain address.

After a thorough analysis of the above factors, let’s check the side functions of the program, that is: all internal and external links, meta robots, duplicate domains, etc. Below are the other Screaming Frog options:

– page depth level,
– file size,
– inlinks,
– outlinks,
– XML Sitemap Generator,
– pictures,
– cannonical,
– 301 or 302 redirects,
– User-Agent Switcher,
– flaws with urls,
– Anchor text.

It is also worth mentioning the possibility of analyzing file sizes, errors (e.g. 4xx, 5xx), headers, as well as problems with URLs, which include, for example, underlines, capital letters or even the lack of ASCII characters.

We can significantly speed up the work in Screaming Frog by performing actions in the right order, taking into account individual assumptions specified in the initial SEO strategy. If we want a comprehensive on-site website optimization – this tool will turn out to be perfect, however, we have to buy the premium version, because the free version is too limited to perform a meticulous domain analysis.

How to crawl problematic sites with Screaming Frog?

Crawling problematic sites doesn’t have to be difficult. All you have to do is go to the “Advanced” tab, and then change the settings of several important options, including “Always follow redirects”, “response timeout” as well as “max redirects to follow”.

Always follow redirects
After selecting this option, Screaming Frog will pay attention to 3XX redirect directives, and what’s more – he will ignore the so-called crawl depth. The above functionality is very useful when we intend to strongly change the structure of the website or migrate to other URLs.

Remember!
Thanks to the “always follow redirects” option, we will verify multiple redirects and the redirect loop much more accurately than in the default settings. This is because SF often does not identify all “deep redirects”.

If, however, we want to check incorrect redirections within the website – we recommend downloading the “Redirect Chains” report from the Reports tab. We will then check the URLs redirected more than twice (single redirects are not visible here).

Response timeout
In the event that we have a free domain, but we want to obtain as much relevant data as possible about it, we should increase the waiting time by Screaming Frog for an HTTP response. We recommend that you change the value beyond the default of 20 seconds.

Max redirects to follow
An interesting option that allows you to direct Screaming Frog robots to determine the length of the redirect path that they are to present in the summary. As a standard, crawlers follow 5 consecutive redirections.

This tab offers one more additional function – extracting images with the srcset attribute. The use of this option is very useful when we want to place the same image in different resolutions – for example in responsive websites.

Summary

Screaming Frog, developed in 2010 (founder: Dan Sharp), is used all over the world by the vast majority of SEO specialists. It has many useful functions, and its main operation is based on reliable scanning of websites, so that you can later verify their dysfunctions and effectively fix them. At Paraphrase-Online, we use Screaming Frog primarily to improve on-site SEO audits and we believe that in this aspect it is an absolute leader!

FAQ

Here are the answers to the most frequently asked questions about Screaming Frog SEO spider!

When is it worth analyzing the website?
The analysis of the website is very useful when we see that it is not working properly, its position does not increase according to the assumptions made, and the traffic from SERPs has definitely decreased. It is then worth verifying meta tags, headers, internal and external links, as well as other on-site factors available in Screaming Frog, which may be a potential cause of irregularities.

What is the price of Screaming Frog?
Screaming Frog is priced at $ 149 annually. The premium version is free, but it has many limitations (e.g. no possibility of integration with Google Search Console and Google Analytics).

Functionality: Whether from a small screen we read differently than from a large screen

Articles comparing reading from large and small screen present conclusions very similar to those that combined reading from the screen and paper. This time it’s not a piece of paper, but the big screen is the place where the reader’s eye slows down. He rushes on the screen of the smartphone.

Reading from a large screen seems to be more insightful and pleasant, while reading from a smartphone is supposed to be more superficial and requires more effort.

What does this mean for people who write web content? Do you have to write even more succinctly and simply? Although the answer “yes” seems intuitive, the research clearly does not confirm it.

Yes, the attention of the person reading the text from a small screen is usually very dispersed. People often rush using smartphones, even if they do not have to do it, because the vast majority of mobile searches are done not on the way, but … at home. Even so, the average session time is twice as short when reading from a small one rather than from a large screen.

So you have to write not shorter, but rather with breaks in mind:

– what is the most important to put in the beginning,                           

– divide the text into short paragraphs (consisting of 1-4 sentences),

– insert subheadings,

– summarize.

These are not new writing rules, but their importance is increasing. When we write for mobile users, we need to focus on what is most important. For example, I was tempted to enter the average session time on a smartphone in one of the previous paragraphs, but I decided that this is a detailed information that an interested person can complete by clicking the link.

When reading from a small screen, the memory is overloaded. A small fragment is displayed on the screen, so more often you need to refer to the saved context. There are few elements in short-term memory (according to George Miller there are 7 of them) and disappear from there after 20 seconds.

We must therefore write as clearly as possible, avoid ambiguity where it is unnecessary. For example, imagine that you find a course offer on the website called Designing pages for cavemen.

Hmm … it sounds interesting, but also weird. Is the “caveman” the definition of the client who would buy the course, or rather the recipient of the site, which the student will be able to design? In the first case, the customer could feel offended. In the second one, he might think that designing such a website is a very simple task, so it may not be worth buying a course.

Meanwhile, the author wanted to encourage the design of pages well-absorbed for the human mind. And the mind has not changed so much over the last 10,000 years (that is, from the time of cavemen). The title of the course will be more precise and simpler: Functionality and the human mind. How your clients think.

Despite the limitations that a small screen imposes on readers, there is no need to simplify the texts by force. Understanding the text read from the small screen does not fall, and can even grow.

Participants in one of the studies had to read articles on various topics using a telephone or computer. Half of the texts were easy, half – difficult, and users had access to both devices on both devices. After reading, they received a multiple-choice text, which helped to assess how well they understood the texts.

Interestingly, the level of understanding was slightly higher (by 3 percent) on mobile devices. Some respondents emphasized that they prefer to read on small screens due to finger scrolling or a smaller number of diffusers.

However, when reading from a small screen, we pay the price in the form of reading speed. It is the time of reading an article divided into its length, counted in words. With easy articles, the speed was similar, but readers slowed down when they faced harder text on a small screen. They spent over 30 milliseconds more over each word.

To achieve the same level of understanding as on a larger screen, the subjects had to read more carefully and try to remember the most important information or to return to the fragments already read.

So you do not have to simplify the easy-to-read text to read it well from the small screen. However, you should devote a lot of attention to difficult texts, such as: regulations, report or privacy policy. Thanks to the principles of simple writing, even the most intricate text can be easily formulated.

SEO: What makes mobile search stand out

In times when more searches are made from phones than from computers, and Google, regardless of which device you use, displays the results from your mobile index (Mobile-First Index), you should think about SEO in the context of smartphones.

A quick search for a goal is a distinguishing feature of mobile search. Big screens have been with us since the beginning of the Internet, after which then it was mostly surfing, so it wandered for pleasure, by accident making interesting discoveries.

We also use the smartphone to pass the time, but the analysis of phrases entered into the search engine shows that most of our inquiries have a clear purpose. They help in making decisions, also when we need it immediately, because we are in a shop or at the airport. In addition, such searches translate into actions very quickly, even within an hour.

So let’s get to know the logic of mobile search in order to reach the reader at the moment when he / she is most interested in our knowledge / service.

Smartphones dominated the local search (with the extremely popular phrase near me in the English-speaking world), but not only. They also mastered the first stage of the search. When we want to learn something, we enter the first questions in the search engine on the phone. Also when we are at home, because it is convenient, fast and does not require a computer. But further intensive searches are often carried out on a larger screen.

A good example is travel planning. For mobile search we are more likely to enter the phrase flights to than flights from. A person entering flights may not yet be sure where or when he wants to fly, and someone who is planning a flight with probably already made a decision or at least narrowed the options and prefers to book on a large the screen.

Phrases entered into the mobile search engine often take the form of questions, starting with: how, where, or, for example, Where to buy organic vegetables, Is the soy sauce is vegan, How to cook millet groats. Thus, titles or subheadings that start with these words will match the search queries.

Mobile phrases tend to be longer than those typed on larger screens, but they certainly contain shorter words, e.g. food instead of restaurant.

Smartphones are the most personal of all devices and so is the search. We are looking for information that is important from our point of view. Over the ocean, the meaning of phrase for me has increased (by 60% in two years), for example car insurance for me, which dog is good for me.

The phrases from I should also be popular, eg Which dog should I choose? Tools that meet these queries are insurance comparison websites or quizzes that help you choose a dog breed.

Please search engine for advice that we would ask friends for (eg comfortable climbing shoes). We try to be precise in your queries to get useful results as soon as possible, e.g. shampoo for dyed hair.

We want answers NOW. It’s not enough that we have it under the fingertips. We raise the bar higher. That is why search for the best word has increased significantly, for example the best toothpaste. Thus, the division into products requiring greater and lesser attention has disappeared. Customers want to be sure of every decision they make, small or large. Therefore, they expect advice, ideas and inspiration.

Copywriting: How to describe links to raise the ranking

The third part of the press will be devoted to an inconspicuous subject, which is very important for both the reader and the search engine. This is the anchor text, whether in the polonized version of the anchor or the content of the link. Highlighted words that we click.

According to an accurate metaphor, Ireneusz Iwański, the descriptions of the links are like signs showing the possibility of exiting the motorway. They send signals to both the reader and the search engine.

The role of links does not end with the fact that they help the reader gain credible information from another side. Thanks to links, Google may consider your website to be:

– consistent with the intention of the reader,

– current,

– deserving a higher place in the ranking.

However, for the inconvenient link to do all of this, it must be accurately described. Suppose I want to put a link to an article Do not be afraid to write straight. What science research says about the benefits of affordable style.

The simplest option will be a “naked” URL, that is: https://www.paraphrase-online.com/blog/webwriting/synonyms-a-good-way-to-effective-web-positioning/. At first glance, you can see that such a link introduces visual confusion because it is not embedded in the text. It also gives a bad testimony about the author’s technical knowledge.

Let’s try differently then. To learn more about the rules of simple writing, click here. In many articles we find a clue to avoid such terms as click, check, read more. Why? Research on the functionality of websites unambiguously confirms that users are interested in effectiveness: they want to reach interesting content in the shortest possible time.

The reader who sees the words click here does not know what he will find on the other side of the link and is it worth waiting a few seconds for the new page to load. It starts to hesitate. And hesitation gives you the boost to click the Back button.

Yes, the first part of the sentence heralded what the article will be about (To learn more about the rules of simple writing …), however, this link stands out in color and catches the eye, so it is worth making it understandable regardless of the surrounding content, e.g. Simple writing can even make the most intricate text formulate easily.

What about click here? Sometimes you can leave them. It is a call to action and it will be useful when we want to draw the reader’s attention to any particularly useful and reliable source. Of course, you can refine such a request, eg Learn more about writing in simple language.

Do key phrases in link descriptions mean? It is worth using them, but they do not have to sound exactly as the search engine or phrases planning tools suggest. In our example, a precise keyword phrase would be a simple language. However, such a description of the link would be too poor to convey to the reader what the article is about.

Google hints and related phrases that appear under the results, for example how to write comprehensibly, can also help. The most important thing is, however, that the words you use are naturally associated with the text and are closely related to the topic. If only your link descriptions accurately describe the content to which you are sending, they will send a good signal to Google.

And how do the links influence the position in the ranking? After all, if you refer the reader to another source, you promote that page, not your own. Yes, but you can also post internal links in the article, that is, those that refer to your other texts.

Thanks to internal links, you set the paths by which readers move and give them an excuse to stay on the site. It’s a simple and yet very effective SEO technique thanks to which:

– the duration of the session is prolonged,

– the bounce rate decreases,

– the user browses more pages during one session.

All of these benefits will additionally mean that Google will consider the page to be compatible with the intention of the reader. For a search engine, compliance with the intention is even more important than compliance with the topic. Sometimes it happens that results consistent with the topic do not give the user what they are looking for.

Internal links have a good effect on page indexing. If the articles are closely related to each other, Google’s spider will be easier to follow the content you publish. So post links to at least a few of your articles in each post. You will gain something else in this way. Your older articles will be treated by the search engine as still valid.

Google appreciates the news and links leading from newer articles to older ones give the latter freshness. Those who follow your internal links refresh the content, showing that it is still up-to-date. Additionally, if you update your old article by adding some new paragraphs to it, Google will see it and crawl it again, which will probably raise its position.

Internal links also have an effect on something that is particularly valuable in SEO, i.e. links that lead to your site from other websites. An incoming link (backlink) provides Google with knowledge about what your site is about.

The more people put a link to my page with a description of a simple language, how to write comprehensively or anything similar, the more certainty a search engine will gain, that my site should take the place in the ranking for this phrase.

A large number of links from other websites that lead to your site is more important to Google than matching key phrases in the title or title, article length or page load time. Despite the changeable position of Google itself, research confirms that more reliable inbound links decide about a higher place in the ranking.

And what influence do internal links have on incoming links? Most people recommending your site probably post a link to the home page. The value of such a link is not as high for the search engine as the subpages link, e.g. those with your articles. By creating a close network of internal links, you increase the chance that other authors will send their readers to different sources on your site, not just to the home page.