Jump to content
Sign in to follow this  
yukon

Live Backlink Checker

Recommended Posts

This is a live backlink checker that I'm working on.

 

It's a Windows desktop app., it's very minimal right now. It downloads multiple pages in parallel (threading) so there's no bottleneck for web pages that are slow, the rest of the sites/pages will keep downloading instead of waiting. 

 

I went ahead & added a column in the data grid for Wayback.

 

I have it setup to export the data as a CSV file.

 

The URLs in the screenshot are all test data (list of restaurants) looking for youtube.com.

 

 

 

 

 

 

1122.png

Share this post


Link to post
Share on other sites

SEOPress WordPress SEO plugin

Now if you could get it to integrate with Majestic and Moz and pull their metrics for each link page... Then have it grab anchor text.... hmmmm

 

Doubtful that will happen.

 

The way I understand those APIs is the developer has to pay out of pocket monthly & most payment tiers have limited downloads regardless of the 3rd party app. profit. A developer has no idea how many sales they'll make monthly, so If they don't make sales one month then everyone that already payed for the app. would lose the API service (unless the developer is Bill Gates).

 

I don't see how those APIs are profitable for 3rd party developers unless the 3rd party app. was billed monthly on an already popular app. to recoup an API cost.

 

To be honest I'm not a fan of APIs because when they break (they always break (eventually)), the 3rd party app. developer gets blamed when it's out of their control.

 

The app. currently imports links but each end user has to pay their own bill for generating their own link profiles. IMO that's only fair.

Share this post


Link to post
Share on other sites

What is it importing links from?

 

Yes, I agree about the API situation. You would really have to build a solid subscriber base before implementing the API (or just have money to burn). With so many backlink tools on the market, it is tough to feel secure that you are going to get a piece of the pie to make the API cost worthwhile.

Share this post


Link to post
Share on other sites

What is it importing links from?

 

Yes, I agree about the API situation. You would really have to build a solid subscriber base before implementing the API (or just have money to burn). With so many backlink tools on the market, it is tough to feel secure that you are going to get a piece of the pie to make the API cost worthwhile.

Right now I'm importing links from a text file, one URL per line.

 

I'll probably add an option to import from a CSV file. Still If someone had an existing CSV file with URLs all they would have to do is copy a single URL column from a CSV & paste into a text file.

 

I'm already exporting as a CSV file.

 

Also, I do plan on adding anchor-text to the app., maybe even check to see If link found is from a hyperlink or an image (alt-text).

Share this post


Link to post
Share on other sites

When will this be availible for us ;)

 

I would like to make a demo version of this link checker app. but I have a lot of cleanup to do first. When building apps. for other people to use you have to do a lot of error handling to keep the app. running smooth.

Share this post


Link to post
Share on other sites

I ran a speed test to see how fast I can download webpages & scrape the offline file for a target URL. I downloaded & parsed 295 pages/backlinks in exactly 1 minute.

 

Not bad considering my internet service blows (AT&T DSL). Seriously, my internet service is intermittent, drops connections all the time & bogs down very easily.

 

If someone here has a very fast internet connection I would be interested in maybe later on having them beta test this Windows desktop backlink checker app. to see what kind of speeds they get. I still have a lot of work to do before I'm comfortable releasing this to anyone else.

 

Just throwing this out there If anyone might be interested later on when the app. is ready.

 

Also, If anyone is running any other live link checkers, how's my app. speed compare to what you're running, am I slow, fast, the same?

Share this post


Link to post
Share on other sites

I'd be interested in testing it. ATT Uverse here (I have uverse at work but it's stuck at 1.5mbps dsl speed because there is no fiber to my building ... something I wish I knew BEFORE signing the lease).

 

29 down, 4 up.

Share this post


Link to post
Share on other sites

I would be happy to test anything for you. I have whatever the fastest cable offering is that Comcast provides. I can check the exact speed numbers.

 

As a comparison, I would say it sounds slightly slower than what I am seeing with Backlinks X-Ray right now, but not far off. It is not a true apples to apples comparison though as Backlinks X-Ray is extracting more data right now.

Share this post


Link to post
Share on other sites

I'd be interested in testing it. ATT Uverse here (I have uverse at work but it's stuck at 1.5mbps dsl speed because there is no fiber to my building ... something I wish I knew BEFORE signing the lease).

 

29 down, 4 up.

 

Thanks,

 

I'll keep this thread updated & let you know when the app. is ready for testing.

Share this post


Link to post
Share on other sites

I would be happy to test anything for you. I have whatever the fastest cable offering is that Comcast provides. I can check the exact speed numbers.

 

As a comparison, I would say it sounds slightly slower than what I am seeing with Backlinks X-Ray right now, but not far off. It is not a true apples to apples comparison though as Backlinks X-Ray is extracting more data right now.

 

Thanks, I'll keep you updated.

 

One other thing I forgot to mention is no two users will ever have the same speeds on this type of app because the PC processor & memory are also factors along with internet speed.

 

The way this app works is, I load a list of URLs into a Listbox (left side of OP screenshot), the URL list is all put in memory (PC memory) at one time when the start button is clicked. The app. takes the list of URLs in memory & starts downloading all the pages as a plain text file to the offline PC desktop, I did this for verification that each downloaded webpage matches the data grid row number (ex: file#1, file#2, file#3, etc...) while testing.

 

The list of URLs are on multiple threads which means fast loading pages will download out of order compared to the Listbox URL list, basically passing the web pages that might take a longer to download for things out of my control (ex: slow server, large file sizes, etc...).

 

I parse the target domain with regex as each web page is being downloaded.

 

Just thought I should mention that.

 

I have a slow internet connection because I'm located about 10 miles outside of the nearest town. I'm probably lucky to even have DSL. My PC is probably average as far as processor & memory, again, others will probably have higher speeds (I hope) because of better hardware & faster ISP.

 

Here's my stats:

 

  • Download speed: 1.29 Mbps
  • Processor: 2.5 GHZ
  • PC Memory: GB

Share this post


Link to post
Share on other sites

I  used to build computers for fun, and was part of a computer forum, then there were some shenanigans and about 10 of us splintered off into a private forum (mostly dead after 10 years) and one of them worked for intel. Every few years he'll offer CPUs that he gets trays of. $60 for a $1000 cpu basically. This one is a couple years old but still a Xeon E3-1285 v2 3.6ghz, 4 core, 8 threads, 16gb of ram. 

 

My cheap ass should get an SSD so it'll really fly.

Share this post


Link to post
Share on other sites

I  used to build computers for fun, and was part of a computer forum, then there were some shenanigans and about 10 of us splintered off into a private forum (mostly dead after 10 years) and one of them worked for intel. Every few years he'll offer CPUs that he gets trays of. $60 for a $1000 cpu basically. This one is a couple years old but still a Xeon E3-1285 v2 3.6ghz, 4 core, 8 threads, 16gb of ram. 

 

My cheap ass should get an SSD so it'll really fly.

 

That's cool, glad to have a hardware guy on the forum.

 

I'm about ready to send you & Mike a PM to download the Live Link app. exe. All this preventative maintenance (error handling) is a lot of work, lol.

 

I've only tested this on Win7 64bit.

Share this post


Link to post
Share on other sites
 

 

 

Alright, I ran it a few times. I'm getting through 250 links in about 20 seconds. There are often a few stragglers though that it will take another 10-15 seconds to complete.

 

I'm on Windows 8.

 

AMD FX-6200 Six-Core Processor 3.80 GHz

10 GB Ram

 

According to http://www.speedtest.net/, my download speed is 92.45 Mbps and upload speed is 12.13 Mbps.

 

 

This is a PM Mike sent me with his permission to copy/paste here on the forum.

 

Mike, thanks for the results.

 

This is what I thought would happen, your hardware & internet speed are way faster than mine & your downloading webpages 2.5 times faster than I am (250 pages per min. vs. 750 pages per min. average). That's good.

 

I'm not sure If I can do anything about the few URLs/pages lagging behind. Like I mentioned earlier in the thread the app. is downloading a bunch of pages all at once so the faster loading web pages should be downloading first. It's the same as If you open 20 tabs (example) in your regular web browser, some pages will load faster than others, some pages will load slow (server speed, large web page file size, etc...).

 

My question is, does the app speed compare to similar apps? I think you've mentioned testing a few apps that check live backlinks.

 

Thanks.

Share this post


Link to post
Share on other sites

Got baby duties this AM, but I'll run it today for sure.

 

Damn MikeF that ISP speed is nasty!

 

Still pissed i'm 1.5mbps at work. Ugh.

Share this post


Link to post
Share on other sites

Got baby duties this AM, but I'll run it today for sure.

 

Damn MikeF that ISP speed is nasty!

 

Still pissed i'm 1.5mbps at work. Ugh.

 

Thanks for the help.

 

Yea, that 92.45 Mbps download speed is way faster than what I have.

Share this post


Link to post
Share on other sites

 

My question is, does the app speed compare to similar apps? I think you've mentioned testing a few apps that check live backlinks.

 

 

Backlinks X-Ray does the same amount of links in about 1 minute and 20 seconds. As I said before though, it is not really a true apples to apples comparison as Backlinks X-Ray is pulling a lot more data. It is definitely faster though.

Share this post


Link to post
Share on other sites

Backlinks X-Ray does the same amount of links in about 1 minute and 20 seconds. As I said before though, it is not really a true apples to apples comparison as Backlinks X-Ray is pulling a lot more data. It is definitely faster though.

 

Thanks, I appreciate the feedback.

 

I see what you're saying about Backlinks x-ray but from what I've read on the Majestic API the data is returned via an xml or json file, I'm guessing it's a single file per each API (Majestic & Moz). I haven't used the API so obviously Bxray knows more on that subject since it's built into their app.

 

The API works over the HTTP protocol, taking parameters from the request and returning a response in either XML or JSON.

 

 

I'm happy the app speed is at least in the same ballpark as similar apps. That gives me motivation to maybe do something more with the app.

 

Like I said earlier bottlenecks on the app will never be the same for all users because everyone is running different hardware & different download speeds. I mean the worlds fastest PC will still bog down with a 56K dialup internet connection. An app is only as fast as it's weakest point (hardware, internet speed).

 

Once the webpages are downloaded the app will process the offline data at lightening fast speeds, even on my PC (lol).

 

I have a few ideas, I think the speed test are decent enough to move forward. I didn't want to get too far along & find out other users were having speed issues.

 

If anyone else would like to test the app, let me know.

Share this post


Link to post
Share on other sites

Oh no, Backlinks X-Ray is not pulling Majestic data. It has the option to pull Moz, but I did not have that running. That is an extra step. Moz would slow the results down a lot because the free API only allows 10 URLs every 2 seconds or something like that.

Share this post


Link to post
Share on other sites

Just an update.

 

I added a few more columns to the data grid:

 

  • Row Number
  • Backlink URL
  • Links Back
  • Page Title
  • Page Size (KB)
  • Cache (Google Cache text version)
  • Date (when live link was checked)
  • Time (when live link was checked)
  • Wayback

I'll need to do some error handling for problem pages with multi-line page titles, example:

 

<title>
       Captain D's - Your Seafood Restaurant | Home
</title>

So far I'm doing everything on the fly while pages are being downloaded, the speed seems to be about the same as before I started adding on more data parsing.

 

 

 

 

 

 

 

 

221.png

 

Share this post


Link to post
Share on other sites

Not sure but I'm having trouble with mine. First uploaded a 2,000 link file and it took forever, I stopped it at 1,000 and it was like 6 minutes in.

 

So I dropped in a 400 link file and it started off fairly strong, 35 seconds to 200 but then it stalled. Did nothing, so I clicked start again and it jumped to 352 links and then died. Wouldn't finish the file even after clicking start again. 

Share this post


Link to post
Share on other sites

Not sure but I'm having trouble with mine. First uploaded a 2,000 link file and it took forever, I stopped it at 1,000 and it was like 6 minutes in.

 

So I dropped in a 400 link file and it started off fairly strong, 35 seconds to 200 but then it stalled. Did nothing, so I clicked start again and it jumped to 352 links and then died. Wouldn't finish the file even after clicking start again. 

 

Thanks for the feedback.

 

I think I'm going to need to overhaul the threading so each user has control over the number of threads running at the same time. I've noticed the app downloads webpages very fast on smaller list of URLs which tells me it's bogging down somewhere on the threads.

 

So... looks like allowing each user to select the number of threads based on their own setup (PC hardware & ISP speed), also maybe a timeout after a few seconds for problem web pages/URLs, keep track of each problem page download time, add those to another list & prompt the user to try & run problem pages a 2nd time If needed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.