Scrape / eavesdrop AJAX data using JavaScript?

Question

Is it possible to use JavaScript to scrape all the changes to a webpage that is being updated live with AJAX? The site I wish to scrape updates data using AJAX every second and I want to grab all the changes. This is a auction website and several objects can change whenever a user places a bid. When a bid is placed the the following change:

The current Bid Price The current high bidder The auction timer has time added back to it

I wish to grab this data using a Chrome extension built on JavaScript. Is there a AJAX listener for JavaScript that can accomplish this? A tool kit? I need some direction. Can JavaScript accomplish this??

Answer 1

I'm going to show two ways of solving the problem. Whichever method you pick, don't forget to read the bottom of my answer!

First, I present a simple method which only works if the page uses jQuery. The second method looks slightly more complex, but will also work on pages without jQuery.

The following examples shows how you can implement filters based on method (eg POST/GET), URL, and read (POST) data and response bodies.

Use a global ajax event in jQuery

More information about the jQuery method can be found in the documentation of .ajaxSuccess. Usage:

jQuery.ajaxSuccess(function(event, xhr, ajaxOptions) {
    /* Method        */ ajaxOptions.type
    /* URL           */ ajaxOptions.url
    /* Response body */ xhr.responseText
    /* Request body  */ ajaxOptions.data
});

Pure JavaScript way

When the website does not use jQuery for its AJAX requests, you have to modify the built-in XMLHttpRequest method. This requires more code...:

(function() {
    var XHR = XMLHttpRequest.prototype;
    // Remember references to original methods
    var open = XHR.open;
    var send = XHR.send;

    // Overwrite native methods
    // Collect data: 
    XHR.open = function(method, url) {
        this._method = method;
        this._url = url;
        return open.apply(this, arguments);
    };

    // Implement "ajaxSuccess" functionality
    XHR.send = function(postData) {
        this.addEventListener('load', function() {
            /* Method        */ this._method
            /* URL           */ this._url
            /* Response body */ this.responseText
            /* Request body  */ postData
        });
        return send.apply(this, arguments);
    };
})();

Getting it to work in a Chrome extension

The previously shown code has to be run in the context of the page (in your case, an auction page). For this reason, a content script has to be used which injects (!) the script. Using this is not difficult, I refer to this answer for a detailled explanation plus examples of usage: Building a Chrome Extension - Inject code in a page using a Content script.

A general method

You can read the request body, request headers and response headers with the chrome.webRequest API. The headers can also be modified. It's however not (yet) possible to read, let alone modify the response body of a request. If you want this feature, star https://code.google.com/p/chromium/issues/detail?id=104058.