tag:blogger.com,1999:blog-41353071795388380102011-11-27T16:05:49.623-08:00This blog is mainly for developers who are searching for materials that help them doing Web Crawling, I'll try to list everything needed for that. examples and tools illustration will be providedMostafa Sirajhttp://www.blogger.com/profile/18276443600709227885noreply@blogger.comBlogger21100tag:blogger.com,1999:blog-4135307179538838010.post-85217163230972645632009-05-03T01:05:00.000-07:002009-05-03T01:16:38.790-07:00I made a session about "Online Business" and "Custom Search Engines" for CuttingEdge club, the attendees were really interactive during the whole session, unfortunately I didn't have the opportunity to know the feedback of the session from them hopully if you attended the session and viewing my blog now you'll leave me a note :)<div><br /></div><div>here is the material</div><div><br /></div><br /><div><br /><h3>Online Business</h3><br /><iframe src="http://docs.google.com/EmbedSlideshow?docid=dcn76rt7_943dfv7v3fw" frameborder="0" width="410" height="342"></iframe><br /><br /><h3>Custom Search Engines</h3><br /><iframe src='http://docs.google.com/EmbedSlideshow?docid=dcn76rt7_957fd5dg4gt' frameborder='0' width='410' height='342'></iframe><br /></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/4135307179538838010-8521716323097264563?l=crawltheweb.blogspot.com' alt='' /></div>Mostafa Sirajhttp://www.blogger.com/profile/18276443600709227885noreply@blogger.com4tag:blogger.com,1999:blog-4135307179538838010.post-647333975919138992008-09-29T05:53:00.000-07:002008-10-22T06:24:25.797-07:00<span style="font-weight: bold; color: rgb(51, 102, 255);font-size:180%;" ><br />Why this post is so important?</span><br /><script type="text/javascript"<br />src="http://pagead2.googlesyndication.com/pagead/show_ads.js"><br /></script><br /><script type="text/javascript"><br />var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");<br />document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));<br /></script><br /><script type="text/javascript"><br />var pageTracker = _gat._getTracker("UA-5492887-2");<br />pageTracker._trackPageview();<br /></script><br /><p><br />Actually I'm not going to tell why do we need to learn web crawling or how much stuff that could be done and money that could be gained if you can develop a good crawler, because I believe that since you are here then you need to crawl and since you are here it doesn't mean that I'm one of the best bloggers around here but it means that THERE IS NO HELP IN THIS TOPIC. that's why this blog is here because people need web crawling and they can't do it because there is no help. I started searching for some material -just like you- and I found almost nothing so I started to do it by my own and here I am publishing what I know for everyone.</p><span style="font-weight: bold; color: rgb(51, 102, 255);font-size:180%;" >What are my resources</span><span style="font-size:180%;"><span style="font-weight: bold; color: rgb(51, 102, 255);">?</span></span><br /><br />as I said there aren't many resources, however I list here the few resources I found and used<br /><br /><p><span style="font-weight: bold; color: rgb(0, 0, 0);font-size:130%;" >1-Book: "<a style="color: rgb(0, 0, 0);" href="http://www.heatonresearch.com/book/http-programming-csharp.html">Http Recipes For C# Bots</a>" by Jeff Heaton</span><br /><br />This is maybe the only book that talks about Web Crawling from a developer point of view, however I believe that it doesn't go deep enough in order to push you to the real work. I read that book several times and always was able to access his website by crawlers I wrote, however I wasn't able to access other websites because as I stated earlier the book doesn't go deep enough</p><p><br /><a style="color: rgb(0, 0, 0);" href="http://www.ideabubbling.com/Article1.aspx"><span style="color: rgb(0, 0, 0);font-size:130%;" ><span style="font-weight: bold;">2-Web Article: Tools for access site .NET</span></span></a><br /><br />This is maybe the only post I found that talks about real crawlers development, it list a great tools for crawler development, beside describing one crawler that logins to Yahoo Address -with source code included- however in a very superficial way. no description about using the listed tools, no code snippets were used. I use most of the tools listed at that post and they are all very beneficial, I'll try at my blog to list each tool and how to deal with in more details. in order to get you on the road very fast</p><br /><a style="color: rgb(0, 0, 0);" href="http://www.beansoftware.com/NET-Tutorials/Web-Crawler.aspx"><span style="font-weight: bold;font-size:130%;" >3-Web Crawler, spider, ant, bot... how to make one?</span></a><br /><p>Another interesting article that gives a complete crawler example in VB.NET, again I don't think the article can help you write your own crawler for your own purpose but reading it is still beneficial as there isn't a lot of resources like I stated before<br /></p><p><br /><span style="font-weight: bold; color: rgb(0, 0, 0);font-size:130%;" ><a href="http://www.getafreelancer.com/users/feedback_354604.html">4-My own experience</a></span><br /><br />I work as a Software Engineer in <a href="http://www.itworx.com/">ITWorx</a> besides being a <a href="http://www.getafreelancer.com/users/354604.html">freelancer</a> I developed several crawlers for many websites and here is a list of the recent crawlers I made<br /><br />1-Yahoo Answers Crawler<br />2-People search crawler that is gathering information from<br /></p><ul><li>Yahoo People</li><li>Lycos People Search</li><li>Peopledata.com</li><li>Superpages.com</li></ul>3- <a href="http://www.getafreelancer.com/projects/PHP-ASP/Find-Script-Injections.html">Script Enjection finder</a> in which the crawler scan list of websites for injected scripts -Cross Site Scripting-<br /><br />and lots of other crawlers that I built -you can check my recents freelancing projects <a href="http://www.getafreelancer.com/users/feedback_354604.html">here</a>-, I believe that sharing my experiences in this field will be very benificial to you -that's why I'm writing this blog-<br /><p></p><span style="font-weight: bold; color: rgb(51, 102, 255);font-size:180%;" >What are the tools needed to write your own crawler?</span><br /><br />actually I recommend all the tools listed at "<a href="http://www.ideabubbling.com/Article1.aspx">Tools for access site .NET</a>" beside another few tools<br />here they are for completeness<br /><br /><div> 1. <a href="http://www.mozilla.com/" target="_blank">Mozilla's FireFox</a> -the browser that your bot will simulate-<br /></div> <div> 2. <a href="http://www.fiddlertool.com/" target="_blank">Microsoft's Fiddler</a> -Network analysis tool-<br /></div> <div> 3. <a href="http://www.radsoftware.com.au/">RAD Software's Regular Expression Designer</a> -for extracting important data from web pages<br /></div> <div> 4. <a href="http://www.ccleaner.com/" target="_blank">Piriform's CCleaner </a>-for clearing out your cookies (it's not that important)<br /></div> <div> 5. <a href="https://addons.mozilla.org/en-US/firefox/addon/60" target="_blank"> Mozilla's Firefox Addon: Web Developer</a> -will helps you in analyzing the pages<br /></div> <div> 6. <a href="https://addons.mozilla.org/en-US/firefox/addon/1843" target="_blank"> Mozilla's Firefox Addon: Firebug</a> -actually I didn't use it during my work-<br /></div>beside these tools (my suggsetions)<br />7. <a href="http://www.wireshark.org/">Wireshark</a> -another network analysis tool. I'll state later why I would need two network analysis programs-<br />8. <a href="http://msdn.microsoft.com/en-us/vstudio/products/aa700831.aspx">Visual Studio 2005 or later</a> -I'll primary use C# .NET 2.0 to build my crawlers and I may add later a java version of the crawlers I'll build-<br /><br /><span style="font-weight: bold; color: rgb(51, 102, 255);font-size:180%;" >Finally</span><br /><br />please try to read all the resources and get all the tools I listed above in order to be ready for my later posts.<br /><br />comments are very welcomed for this post or anyone later I'll try to start writing my nexts posts as soon as I can.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/4135307179538838010-64733397591913899?l=crawltheweb.blogspot.com' alt='' /></div>Mostafa Sirajhttp://www.blogger.com/profile/18276443600709227885noreply@blogger.com4