search
Carter Cole LinkedInCarters Twitter PageCarter Cole on Facebook Carter Coles RSS

Sunday, October 24, 2010

Proxies proxies everywhere and all for free

SEO Site Tools has gone into an intensive development kick working to get everything ready for v3.0 framework upgrade and more features like export and reporting as well as more metrics and integration to some other services (Google Analytics, Webmaster Tools), but some of this data cant be gathered by the extensions and IP based API rate limit how much data you can pull, so thats where proxies come in, they let you mascaraed as other IPs from other places... I need a big list of proxies to do the dirty work and by modifying a few open source projects ive built my list to 1505 open proxies in just a couple days. Heres how my dealie works...

First i took an open source uptime monitor and created my own plugin to test HTTP proxies using cUrl and PHP

Heres my proxy checker plugin code...

I found it could only connect out port 80 (firewall i guess) so I built my script to only check those it finds with valid ports

heres a shot of my dashboard
1505 proxies up what what
I changed the code that runs the template so i can pass an API parameter to pull xml or json proxy list and status as well as hacking some code so my proxy scraper could add items (and password protection)... oh heres a sample return from my proxy tester API (returns XML for simple parsing)
That lets me simplexml_load_string to test and see what level proxy it is... they go something like

  1. Elite Proxy, connection looks like a regular client
  2. Anonymous Proxy, no ip is forworded but target site could still tell it's a proxy
  3. Transparent Proxy, ip is forworded and target site would be able to tell it's a proxy
and are graded by how and what headers are returned by the proxy (ill upload my proxy judge code later if anyone wants it) so then to make the process even easier i decided i would use some crowd sourcing techniques, i made a script to scrape proxies from text using regex and then made a page to test proxies... then i test the proxies and add the good ones to my database :) i also found a few proxy lists on google and setup a cron job to run and scrape their proxies every few hours...

if you have any questions feel free to bother me... thats what im here for

Saturday, October 23, 2010

.tk API + Wordpress Mu & cPanel with unlimited addon domains = infinite webspace

So shared hosting is super duper cheap now... like freaking 3$ a month, and with unlimited bandwidth and addon domains you can host a ton of stuff there... it wont be the fastest site on the internet but it works well enough. Now i happen to the cPanel conference and met the guys who run the .tk cctld and they have an API that lets you create any .tk domain for a year for free (with unlimited renewals) and adding this to your shared hosting can let you throw up some test webspaces without paying for a bunch of domain names. So if you add this to the cPanel API you can provision and add unlimited domains and websites. Whats a good platform to run anything on? well wordpress of course, but you dont want to have to do all the work to copy and install all those files (even with unlimited storage) so you use something thats a little awesome... WordPress Mu, it lets you create instances of WordPress and combine with a few plugins you can automagically provision entire new domains with a central WordPress install that you can drive and manipulate with whatever you want. oh ya way fun

so let me show you around my cool little script...

Actually i think thats all i can really show you... but its freakign awesome to have so much webspace to spread my legs out in. oh and
To build your own:
  1. First you install Wordpress Mu
    its kinda weird and some of variables get jankey but its quite impressive overall
  2. Then you install Mu Domain Mapping
    dont forget the sunrise.php
  3. Build a .tk API helper plugin
    I use the domainshare API
    I added the ability to scan and try and renew the domains automatically


    thats a teaser of my plugin ill release soon
  4. Integrate your cPanel to provision domains
    Im using this snippet but ill implement XML soon

I have like 12 sites running on it now all doing different things... im trying my had at getting some traffic to generate adwords money, started a blog for my son Seth Cole and my wife Jodi Cole aswell... i think im going to start my own little WordPress.com kinda thing on .tk domains so if you want a free instance of WordPress to play with let me know :) i may even release a public facing version to the public sometime... (depending on demand)

Comment about your automated scripts and ideas you have for this kind of quick cheap hosting of a large amount of webspace... i mean UGC content value of a folder WordPress mu on a domain would be crazy value to SEO, again if you had you community that could build on it