Hey guys glad you're liking it.
Anyone have a workaround for timeout issues with php? I want to open up the scope of the spider but running it with a depth of a (x pages * y keywords) would really hang this thing.
Suggestions?
Hi Dave,
I just modded your script to use parallel curl like my previous code, and shaved a whole load of time off.
This is the result:
Original: 73.2088 seconds to process 49 urls (keyword=myspace)
Modded: 7.0003 seconds to process 49 urls (keyword=myspace)!!! That's 10x faster.
Modded:
[high=php]
<?
// Setting the variables
if(! defined(ABSPATH)) define('ABSPATH', dirname(__FILE__).'/');
include_once ABSPATH . "Parallel_curl.php";
$s = microtime(true); #timer
set_time_limit(180);
// -------------------GooglePrefix------¦-----query--¦suffix¦-counter--¦
$GooglePrefix = "http://www.google.com/search?q=";
$GoogleCountSuffix ="&start=";
$dontcrawlarray = array('wikipedia.org','google.com','amazon.com');
$query = urlencode($_GET['s']);
$drilldepth = $_GET['drilldepth'];
?>
<h1>Adsense Spider</h1>
<form action="<?=$_SERVER['PHP_SELF']?>">
<input type="text" name="s" value="<?=urldecode($query)?>">
<select name="drilldepth">
<?PHP
for ($i = 5; $i >= 1; $i--)
echo "<option value='$i' " . ($i == $drilldepth ? 'selected' : '') . "> $i </option>";
?>
</select>
<input type="submit" value="Crawl">
</form>
<?PHP // Loop SERPs for given keyword
if ($query == ''){
die("No search term defined");
}
echo "<hr><h2>" . urldecode($query) . "</h2>Looping through SERPs";
ob_flush(); flush();
for ($loop = 0, $stop = ($drilldepth != NULL ? $drilldepth * 10-10 : 30); $loop <= $stop; $loop = $loop + 10) {
$CompleteUrl = $GooglePrefix.$query.$GoogleCountSuffix.$loop;
$res = andstatus( $res. webFetcher($CompleteUrl) );
}
echo "<br>";
ob_flush(); flush();
// Take search results and parse out URLs
echo "Parsing URIs from SERPs";
ob_flush(); flush();
$resultURLs = do_reg($res, "/h3.class=r.*(http.*)\"/U");
echo " [" . count($resultURLs) . "]<br>";
ob_flush(); flush();
foreach ($dontcrawlarray as $url)
$resultURLs = array_ereg_search($url,$resultURLs);
// Take URL results and loop through each to find Adsense code
echo "Digging through URIs for Adsense enabled pages";
ob_flush(); flush();
$s = microtime(true);
$pg = new ParallelGet($resultURLs);
$html_of_all_urls = $pg->html_assoc;
$x = 0;
foreach ($html_of_all_urls as $html_source) {
( preg_match("/google_ad/", $html_source) ? ( $matches[] = andstatus( $resultURLs[$x] ) ) : '') ;
$x++;
}
$textdump = "";
echo "<hr><h3>Adsense Enabled Placements:</h3><ol>";
// Loop through matches and format for review
foreach ($matches as $match => $pURI) {
echo "<LI><a href='$pURI' target='_BLANK'>$pURI</a></LI>";
$textdump .= "$pURI\n";
}
// Dump links into a box for easy C&P
echo "</ol><textarea cols=80 rows=5>$textdump</textarea>";
print "<br />total time: ".round(microtime(true) - $s, 4)." seconds to process " . count($resultURLs) . " urls";
function do_reg($text, $regex) {
preg_match_all($regex, $text, $regxresult, PREG_PATTERN_ORDER);
return $regresult = $regxresult[1];
}
function webFetcher($url) {
// Fetch and return source from $url
$crawl = curl_init(); // The curl library is initiated, the following lines set the curl variables
curl_setopt ($crawl, CURLOPT_URL, $url); // The URL is set
curl_setopt($crawl, CURLOPT_RETURNTRANSFER, 1); // Tells it to return the results in a variable
$resulting = $resulting.curl_exec($crawl); // curl is executed and the results stored in $resulting
curl_close($crawl); // closes the curl procedure.
return $result = $resulting; // Returns source
}
function andstatus($data) {
// Function allows a psudo progress status '....' to be displayed
echo "."; return $data; }
function array_ereg_search($val, $array) {
/* This removes $val from $array if found - used to remove the dontcrawlarray URLs */
$return = array();
foreach($array as $v) {
if(!eregi($val, $v)) $return[] = $v;
}
return $return;
}
?>
[/high]
Don't forget to include the class.
Cheers,
John
