search
Carter Cole LinkedInCarters Twitter PageCarter Cole on Facebook Carter Coles RSS
Showing posts with label API. Show all posts
Showing posts with label API. Show all posts

Sunday, October 24, 2010

Proxies proxies everywhere and all for free

SEO Site Tools has gone into an intensive development kick working to get everything ready for v3.0 framework upgrade and more features like export and reporting as well as more metrics and integration to some other services (Google Analytics, Webmaster Tools), but some of this data cant be gathered by the extensions and IP based API rate limit how much data you can pull, so thats where proxies come in, they let you mascaraed as other IPs from other places... I need a big list of proxies to do the dirty work and by modifying a few open source projects ive built my list to 1505 open proxies in just a couple days. Heres how my dealie works...

First i took an open source uptime monitor and created my own plugin to test HTTP proxies using cUrl and PHP

Heres my proxy checker plugin code...

I found it could only connect out port 80 (firewall i guess) so I built my script to only check those it finds with valid ports

heres a shot of my dashboard
1505 proxies up what what
I changed the code that runs the template so i can pass an API parameter to pull xml or json proxy list and status as well as hacking some code so my proxy scraper could add items (and password protection)... oh heres a sample return from my proxy tester API (returns XML for simple parsing)
That lets me simplexml_load_string to test and see what level proxy it is... they go something like

  1. Elite Proxy, connection looks like a regular client
  2. Anonymous Proxy, no ip is forworded but target site could still tell it's a proxy
  3. Transparent Proxy, ip is forworded and target site would be able to tell it's a proxy
and are graded by how and what headers are returned by the proxy (ill upload my proxy judge code later if anyone wants it) so then to make the process even easier i decided i would use some crowd sourcing techniques, i made a script to scrape proxies from text using regex and then made a page to test proxies... then i test the proxies and add the good ones to my database :) i also found a few proxy lists on google and setup a cron job to run and scrape their proxies every few hours...

if you have any questions feel free to bother me... thats what im here for

Saturday, October 23, 2010

.tk API + Wordpress Mu & cPanel with unlimited addon domains = infinite webspace

So shared hosting is super duper cheap now... like freaking 3$ a month, and with unlimited bandwidth and addon domains you can host a ton of stuff there... it wont be the fastest site on the internet but it works well enough. Now i happen to the cPanel conference and met the guys who run the .tk cctld and they have an API that lets you create any .tk domain for a year for free (with unlimited renewals) and adding this to your shared hosting can let you throw up some test webspaces without paying for a bunch of domain names. So if you add this to the cPanel API you can provision and add unlimited domains and websites. Whats a good platform to run anything on? well wordpress of course, but you dont want to have to do all the work to copy and install all those files (even with unlimited storage) so you use something thats a little awesome... WordPress Mu, it lets you create instances of WordPress and combine with a few plugins you can automagically provision entire new domains with a central WordPress install that you can drive and manipulate with whatever you want. oh ya way fun

so let me show you around my cool little script...

Actually i think thats all i can really show you... but its freakign awesome to have so much webspace to spread my legs out in. oh and
To build your own:
  1. First you install Wordpress Mu
    its kinda weird and some of variables get jankey but its quite impressive overall
  2. Then you install Mu Domain Mapping
    dont forget the sunrise.php
  3. Build a .tk API helper plugin
    I use the domainshare API
    I added the ability to scan and try and renew the domains automatically


    thats a teaser of my plugin ill release soon
  4. Integrate your cPanel to provision domains
    Im using this snippet but ill implement XML soon

I have like 12 sites running on it now all doing different things... im trying my had at getting some traffic to generate adwords money, started a blog for my son Seth Cole and my wife Jodi Cole aswell... i think im going to start my own little WordPress.com kinda thing on .tk domains so if you want a free instance of WordPress to play with let me know :) i may even release a public facing version to the public sometime... (depending on demand)

Comment about your automated scripts and ideas you have for this kind of quick cheap hosting of a large amount of webspace... i mean UGC content value of a folder WordPress mu on a domain would be crazy value to SEO, again if you had you community that could build on it

Friday, August 6, 2010

New SEO API - HTTP Redirect Chain Test lets you enumerate all the jumps

One of the things that you should know about redirects and SEO is that only 301 Moved Permanently if there is more than one redirect then the crawler will use the "weakest" code it saw... You can test the tool below and see how it behaves... after about 5 redirects GoogleBot will stop following them so my script also cuts off around that point. You can put any url you like below and it will show you the destination url with any hops it makes in between

To see code and use tool you need to be on original post


You can see that this default one makes too many jumps and it cuts off but lets talk about formats and how to use the API. It can return a JSON or JSONP (requires attribution link) and will follow up to 6 redirects and return info about each leg of the journey
This is a sample call, it takes 1 required parameter url= which is url to check and an optional callback= parameter if you want a JSONP response. (an object wrapped in function as output)

Ok heres the example API url

http://cartercole.com/dev/api/redirectchain.asp?callback=testCB&url=http://migre.me/TAVJ
and a sample output from its response... it has a original url, a destination and its status code, as well as an array called "chain" that holds each leg of the jump

I created this for use with new version of SEO Site Tools but im releasing it publicly because I dont know of any other SEO APIs avaiable that do this kind of results so use the tool and I hope you find it useful. If you have questions bother me at one of my social profiles... im pretty sure theres hundreds now :)
built with CarterCole.com tech

Sunday, June 27, 2010

Google Reader Subscriber Statistics API

So im working on v3.0 of SEO Site Tools... its going to totally rock... rebuilt from the ground up it now takes advantage of the new abilities of HTML5... things like SQL Db from HTML5 spec to provide result caching and historical data on almost every metric. Tons of useful SEO metrics can be pulled out of Google Analytics data but its often hard to get it without using an external server to proxy the information or having to give your password... but with oAuth Authentication to your your copy of SEO Site Tools gets a secure token to your Google Analytics Data can easy be revoked and all processing and data storage happens right on your computer. But enough promoting here is one of the new data sources im introducing... RSS can bring many return visitors if a blog is regularly publishing so how many subscribers do they actually have? well ive hooked into the unpublished Google Reader API to create my own public API. Now SEO Site Tools or your mashup can use subscriber stats as another data source so the data you see here
can now be used anywhere... its up to you the API now spit out all the info... it returns JSON(P) but i think i may also add XML support later... for xml include output=xml and there is the optional callback= parameter to do JSONP

All code is on original post

]]>
thats going to call the function soMZcb(obj) with the following object as its first parameter
theres a TON of data they return but some of the coolest stuff are chart api urls to show you all the subscriber usage stats as well as total subscribers and when they read

if you have any question feel free to ask me id be happy to help... look for all this new data in SEO Site Tools v3.0 its on its way with even more speed and features than ever!

Wednesday, June 23, 2010

Google Talk Status API... now its easy to style your gtalk badges however you like

Google talk is awesome... and for awhile i had the big stupid badge that would show if i was online or not with a click to chat button...



i wanted to be able to use this again and style it to my liking (im just crazy about my html) so i created my own API to pull the badge and return its info as JSON. i tried to do this with YQL but they cache the requests so i had to do it from my server

so heres the super simple script and how to use it
the api is at

http://cartercole.com/dev/api/gtbadge.asp?tk=[appid]


first your going to need to create a badge... (that will let us query your status from a public location)
next you need to find your badge id. if this is the iframe url for the badge
http://www.google.com/talk/service/badge/Show?tk=z01q6amlqklsogqki1t3ujmofuerp8pbc7hi5vuc0e2pv30c5mrb1tnl3im146nnc3h8f0q19sbism61srg12t5f06guvhhbecj9cgk60i2gbv7sqjdqb18k4893trro8nmuccsbfluh362067l4svbhjgibaiodbh3d1n5e1&w=200&h=60 the bold part is the id (or the tk= parameter). thats the badge id you need to pass to the api as the tk= parameter :) along with the optional callback= parameter so result is returned as callback({Object}) so the script can be used as JSONP (see explanation here) heres an example of what it returns (including optional callback)

you must be on main post to view code


thats about it but before one last thing ill do... heres a quick example of its use.

and heres that script live right here... (i tweaked it a bit for better output) if you have any questions then just click below im usually online :)

Sunday, January 3, 2010

Consuming Google Data APIs with ASP

recently i did a post about a forum spamming attack that I analyzed and included some exports from my Google analytics data...to create this report I used a HTTPhelper i created that wraps the WinHTTP object in ASP that makes it really easy to make external requests and pull data from the Google Data API. Its easy to pull different segments and data if you use the data feed query explorer to create the request url (the second test.url in the code below) then just parse the xml and output as desired... unfortunately i haven't found a good asp oauth (recommend one to @cartercole)

the code is here but i also have included it below in this post...

this post contains code see it here



its easy to pull data from the other APIs but its sometimes hard to find the correct service string tp pass so heres my cheat sheet. im all for helping / suggestions so please leve your comments they are always appreciated (and so are links)

Google APIService name
Google Analytics Data APIsanalytics
Google Apps Provisioning APIsapps
Google Base Data APIgbase
Google Sites Data APIjotspot
Blogger Data APIblogger
Book Search Data APIprint
Calendar Data APIcl
Google Code Search Data APIcodesearch
Contacts Data APIcp
Documents List Data APIwritely
Finance Data APIfinance
Gmail Atom feedmail
Health Data APIhealth
weaver
Maps Data APIslocal
Picasa Web Albums Data APIlh2
Sidewiki Data APIannotateweb
Spreadsheets Data APIwise
Webmaster Tools APIsitemaps
YouTube Data APIyoutube
and here is my HTTPhelper class... it does some cool stuff like build post data from a dictionary object and has all the stuff to do Authentication and setting headers for the request (much easier for me because i forget the methods and have to look them up) so if its helpfull for you please feel free to use it :) woohoo open source!


thanks for reading and i hope this helps you do more with the data APIs and get to your data more easily...