Go CrawlFarm Code

I like to make an inventory of big software packages that I create, here it is for my crawlfarm.

I have added the code to Github

Please check it out. I will be making bug fixes and updating code to be more consistent and better. It's a pretty awesome project that I'm pretty proud of :)

Key Golang features, if you want to view it for example code:

  1. Gob
  2. Concurrent For / Select on Channels
  3. Logging using log package
  4. net.Conn, Server and Client, Dial and Accept
  5. sync.Mutex
  6. encoding/json
  7. time.NewTicker
  8. Synchronous and Asynchronous channels
  9. net/http.Client, CheckRedirect
  10. Regex to find href/src attributes

Feel free to fork and create pull requests :)

blog comments powered by Disqus