Web scraping framework in PHP using jQuery selectors

danny · Jan 14, 2013

https://github.com/dabbamont/domquery

This isn't fully tested and isn't well documented *yet*, but I'm using it as a sample for a few companies that are considering me for engineering positions. If you've done any work with scraping or data mining in PHP you'll see how powerful it is.

I've used this for a lot of scraping and it makes it EXTREMELY easy. I'll keep it short and sweet, so I'll post an example of a scraping class I wrote really quick with it (this is also included in the source).

Using the GoogleSearch class:

PHP:

$search = new GoogleSearch("xml");

$results = $search->getResults();

print_r($results);

The actual GoogleSearch class (built using DOMQuery)

Returns:

Code:

PHP:

class GoogleSearch {
    
    public $keyword = "";
    
    private $results = [];
    
    public function __construct($keyword) {
        $this->keyword = $keyword;
    }
    
    public function getResults($maxResults = 10) {
        
        if (count($this->results) > $maxResults) return $this->results;
        
        $curPage = 1;
        $curCount = 0;
        
        while ($curCount < $maxResults) {
            $curCount = $curPage * 10;
            $this->getPage($curPage);
            $curPage++;
        }
        return $this->results;
    }
    
    private function getPage($num) {
        $start = ($num - 1) * 10;
        $doc = new DOMQuery\Doc("http://www.optimum.net/Search?q=" . urlencode($this->keyword) . "&p=$num");
        
        $results = $doc->find("#websearch > div");
        $current = $start;
        
        foreach ($results as $result) {
            
            $current++;
            $link = $result->find("a");
            
            $this->results[$current] = [
                "keyword" => $this->keyword,
                "position" => $current,
                "title" => $link->text(),
                "url" => $link->attr("href"),
                "description" => $result->find("div")->text(),
                "domain" => parse_url($link->attr("href"))["host"]
            ];
        }
        
    }
    
    
}

Rage9 · Jan 14, 2013

Been something like it around for a while:

PHP Simple HTML DOM Parser

It's really slow and memory intensive though. IMHO stick to xpath.

danny · Jan 15, 2013

Oh yeah I used that a few years ago. This is nothing like that. It compiles the selectors to xpath so it's about as fast as pure xpath, but wraps the DOM objects so you can use it just like jquery.

shawnhag · Jan 15, 2013

very cool

mattseh · Jan 15, 2013

Why is GoogleSearch a class? AFAICT, you are not using any code in multiple places, so it's not to avoid DRY.

It's just one method calling another, this could be done in a lot less code, more clearly, in a simple function IMO.

eliquid · Jan 15, 2013

I dont see any non blocking multi threading... thanks for the post though.

danny · Jan 17, 2013

mattseh said:
Why is GoogleSearch a class? AFAICT, you are not using any code in multiple places, so it's not to avoid DRY.

It's just one method calling another, this could be done in a lot less code, more clearly, in a simple function IMO.

GoogleSearch is just an example, but with the way I do things locally I have my reasons. There'd be nothing wrong with doing it that way, maybe it should be simpler just for the sake of the example.

danny · Jan 17, 2013

eliquid said:
I dont see any non blocking multi threading... thanks for the post though.

Do you think working on this would be worth it? With the way PHP works for me it's always about short running scripts with multiple instances. Even if I did asynchronous HTTP requests with closures being passed as callbacks, I don't think it would ever be beneficial as far as performance goes.

To release this I dumbed down the HTTP requests to a file_get_contents because I have to refactor some stuff, but I'm doing something like ajaxsetup, setting headers, have persistent cookies with named instances, returning different types for JSON, HTML, XML responses, can post, can fill and submit selected form elements and return the new page document object. You see where I'm going.

mattseh · Jan 17, 2013

danny said:
Do you think working on this would be worth it? With the way PHP works for me it's always about short running scripts with multiple instances. Even if I did asynchronous HTTP requests with closures being passed as callbacks, I don't think it would ever be beneficial as far as performance goes.

To release this I dumbed down the HTTP requests to a file_get_contents because I have to refactor some stuff, but I'm doing something like ajaxsetup, setting headers, have persistent cookies with named instances, returning different types for JSON, HTML, XML responses, can post, can fill and submit selected form elements and return the new page document object. You see where I'm going.

Been there, done that.

https://github.com/mattseh/magicrequests

Also does async blah blah blah

Search

Search

Web scraping framework in PHP using jQuery selectors

danny

New member

Rage9

Banned

danny

New member

shawnhag

Click for details

mattseh

import this

eliquid

Serpwoo.com

danny

New member

danny

New member

mattseh

import this