We’re not the only Internet services organization out there that monitors our infrastructure and Internet footprint. It seems as though there is a group of technologists and data-miners out there, who call themselves #Netcraft. They collect a very thick, yet diverse chunk of information, too.
But they are not the only type of company or operator in this space. I just reference their site and service, because they do provide a great deal of other information that honestly, I am considering serving up to my readers at some point. Otherwise, I’d ban them.
I was alerted to their presence recently, for I have a domain that is used mainly for reference purposes by other applications on other domains – for load balancing and high availability purposes. The root of that domain has an auto-forward that directs the browser or spider to the other top-level domain, that the average user will see. When this happens (because it generally, shouldn’t) my code sends me a brief email of what time it occurred, some limited header information, etc.
So, there they were, plain as day, ‘spidering’ out to my site, querying for information. But by no means are they the only players out there doing it. I get several, a day.
[Frankly, I’m flattered….]
I don’t have a problem with this, personally. One, you can’t stop them – completely. If you could, would you? It wouldn’t be a bad idea. Although it’s not a lot of bandwidth – entities like this – and rest assured dozens of others – are doing the same thing.
For example – when search engines crawl through your web site, in order to index and determine relevancy of your web site’s pages – similar things are going on. You can, however, control how often they come back.
I get it – #bigdata. Statistics. Usable business and technical intelligence that companies pay for. I get it.
I’ll tell you a short story my father once told me, and I have applied this story to a number of scenarios. He asked me once… “Would you rather have a million-dollar bill, or a million one-dollar bills?”. Silly me, so young, I would say “a million-dollar bill”. He said – “that’s fine – but a million one-dollar bills is the same thing, right?”
You call for a byte’s worth of resources MY company pays for, that is one thing. You call for a million of them, that’s a megabyte. And an impact however small to my clients who host with me, is something for me to consider.
Ask yourself this… what is the difference between a harmless ping to test connectivity, and a denial of service attack?
And who proactively wants all of this business intelligence, and for what purpose?
We're happy to share this resource that we found. The content displayed on this page is property of it's original author and/or their organization.