Pulling enough results

    my (%groups,@results);

    my $scroll = $es->scroll_helper(
        index => 'my_emails',
        type  => 'email',
        body  => { query => {... some query ... }}
    );

    my $doc;
    while (@results < 10 and $doc = $scroll->next) {

        my $thread = $doc->{_source}{thread_id};

        unless ($groups{$thread}) {
            $groups{$thread} = [];
            push @results, $groups{$thread};
        }
        push @{$groups{$thread}},$doc;

    }

Extracting all documents

    my $scroll = $es->scroll_helper(
        index       => 'my_index',
        size        => 1000,
        body        => {
            query => {
                match => {
                    client_id => 123
                }
            },
            sort => '_doc'
        }
    );

    while (my $doc = $scroll->next) {
        # do something
    }

Very often the something that you will want to do with these results involves bulk-indexing them into a new index. The easiest way to do this is to use the built-in "reindex()" in Search::Elasticsearch::Client::5_0::Direct functionality provided by Elasticsearch.

"new()"

    use Search::Elasticsearch;

    my $es = Search::Elasticsearch->new(...);
    my $scroll = $es->scroll_helper(
        scroll         => '1m',            # optional
        scroll_in_qs   => 0|1,             # optional
        %search_params
    );

The "scroll_helper()" in Search::Elasticsearch::Client::5_0::Direct method loads Search::Elasticsearch::Client::5_0::Scroll class and calls "new()", passing in any arguments.

You can specify a "scroll" duration (which defaults to "1m") and "scroll_in_qs" (which defaults to "false"). Any other parameters are passed directly to "search()" in Search::Elasticsearch::Client::5_0::Direct.

The "scroll" duration tells Elasticearch how long it should keep the scroll alive. Note: this duration doesn't need to be long enough to process all results, just long enough to process a single batch of results. The expiry gets renewed for another "scroll" period every time new a new batch of results is retrieved from the cluster.

By default, the "scroll_id" is passed as the "body" to the scroll request. To send it in the query string instead, set "scroll_in_qs" to a true value, but be aware: when querying very many indices, the scroll ID can become too long for intervening proxies.

The "scroll" request uses "GET" by default. To use "POST" instead, set send_get_body_as to "POST".

"next()"

    $doc  = $scroll->next;
    @docs = $scroll->next($num);

The "next()" method returns the next result, or the next $num results (pulling more results if required). If all results have been exhausted, it returns an empty list.

"drain_buffer()"

    @docs = $scroll->drain_buffer;

The "drain_buffer()" method returns all of the documents currently in the buffer, without fetching any more from the cluster.

"refill_buffer()"

    $total = $scroll->refill_buffer;

The "refill_buffer()" method fetches the next batch of results from the cluster, stores them in the buffer, and returns the total number of docs currently in the buffer.

"buffer_size()"

    $total = $scroll->buffer_size;

The "buffer_size()" method returns the total number of docs currently in the buffer.

"finish()"

    $scroll->finish;

The "finish()" method clears out the buffer, sets "is_finished()" to "true" and tries to clear the "scroll_id" on Elasticsearch. This API is only supported since v0.90.5, but the call to "clear_scroll" is wrapped in an "eval" so the "finish()" method can be safely called with any version of Elasticsearch.

When the $scroll instance goes out of scope, "finish()" is called automatically if required.

"is_finished()"

    $bool = $scroll->is_finished;

A flag which returns "true" if all results have been processed or "finish()" has been called.

"total"

"max_score"

"aggregations"

"facets"

"suggest"

"took"

"took_total"