User Guide

<https://openresty.gitbooks.io/programming-openresty/content/testing/index.html>

Features inherited from Test::Base

Still we would highlight some of the inherited features here for those unfamiliar with Test::Base.

Meta sections

"--- ONLY"

Runs the surrounding test block only. You need to remember removing "--- ONLY" before committing your changes though. Don't worry, the test scaffold would warn you loudly on the console when you left a "--- ONLY" in some test file.

It is also very intuitive for the developer's workflow. One does not have to specify a (unique) test name on the command-line; just find the block in the editor, insert a "--- ONLY" line right away, and run the current test file immediately (for Vim users, the final step is as simple as entering ":!prove %" where ":!" is the Vim way of running an external shell command and "%" would get substituted with the current file being edited in Vim's buffer).

This is definitely one of the most useful and frequently used features.

"--- SKIP"

Skips the surrounding test block unconditionally. You can use "--- skip_nginx" and "--- skip_nginx2" providied by this module (see their documentation below) to conditionally skip tests according to the current NGINX server versions. You can also use "--- skip_openssl" (see its documentation below) to conditionally skip tests according to the current OpenSSL version.

Filters

We can use filters to preprocess the values of our blocks, which can make specifying special values much easier.

For example, we could chop off the last new-line character (if any) of the current section value by specifying the "chomp" filter, like this:

    --- response_body chomp
    Hello world!

Without the "chomp" filter, the value of the "response_body" section would take a trailing new line.

We list some of the common filters below (please keep in mind that one can define custom filters!)

"chomp"

Remove the last character if it is a newline.

"chop"

Remove the last character no matter what it is.

"eval"

Treat the section value as a Perl source code snippet, evaluate it right away, and use the returned value of the Perl code snippet (usually being the value of the last expression).

This is very useful for specifying non-printable characters in section values, as in

    --- response_body eval
    "I don't know what \0 is.\n"
    

run_tests

no_shuffle

use_hup

no_long_string

no_diff

worker_connections

    worker_connections(1024);
    run_tests();

Default to 64.

repeat_each

Default to 1.

shutdown_error_log

For example,

    --- shutdown_error_log
    cleanup resolver

or an example for using an array value,

    --- shutdown_error_log eval
    ["cleanup", "resolver"]

WARNING: skip the shutdown_error_log tests under the HUP reload mode.

no_shutdown_error_log

Here is an example:

    --- no_shutdown_error_log
    [error]

This test will fail when any of the line in the error.log file contains the string "[error]".

env_to_nginx

For example,

    env_to_nginx("foo", "bar=123", "baz=hello world");
    run_tests();

will result in the following lines to be inserted into the resulting nginx.conf file generated by the test scaffold:

    env foo;
    env bar=123;
    env 'baz=hello world';

The latter two are examples of setting values directly to the environments. You can also set values directly on the Perl land, before calling this "env_to_nginx" function, for instance,

    $ENV{baz} = 'hello world';
    env_to_nginx("baz");

If you just want to pass certain environments to a particular test case (or test block), you can just use the "--- main_config" secion directly. For example,

    --- main_config
    env foo;
    env bar=123;

You can check out nginx's official document on its "env" directive below:

<http://nginx.org/r/env>

By default, only the following environments are passed:

• MOCKEAGAIN_VERBOSE
• MOCKEAGAIN
• MOCKEAGAIN_WRITE_TIMEOUT_PATTERN
• LD_PRELOAD
• LD_LIBRARY_PATH
• DYLD_INSERT_LIBRARIES
• DYLD_FORCE_FLAT_NAMESPACE
• ASAN_OPTIONS
• MOCKNOEAGAIN_VERBOSE
• MOCKNOEAGAIN

workers

    workers(2);

Default to 1.

master_on

By default, the master process is not enabled unless in the "HUP reload" testing mode.

log_level

This global setting can be overridden by the per-test-block "--- log_level" sections.

Default to "debug".

check_accum_error_log

no_root_location

bail_out

This function will also do all the necessary cleanup work. So always use this function instead of calling "Test::More::BAIL_OUT()" directly.

For example,

    bail_out("something bad happened!");

add_cleanup_handler

For example,

    add_cleanup_handler(sub {
        kill INT => $my_own_child_pid;
        $my_own_socket->close()
    });

add_block_preprocessor

The processor subroutine is always run right before processing the test block.

This mechanism can be used to add custom sections or modify existing ones.

For example,

    add_block_preprocessor(sub {
        my $block = shift;

        # use "--- req_headers" for "--- more_Headers":
        $block->set_value("more_headers", $block->req_headers);

        # initialize external dependencies like memcached services here...
    });

We can leverage this feature to specify a default value for one or more sections in a single test file. For instance,

    use Test::Nginx::Socket 'no_plan';

    add_block_preprocessor(sub {
        my $block = shift;

        if (!defined $block->config) {
            $block->set_value("config", <<'_END_');
    location = /t {
        echo $arg_a;
    }
    _END_
        }
    });

    run_tests();

    __DATA__

    === TEST 1:
    --- request
        GET /t?a=3
    --- response_body
    3



    === TEST 2:
    --- request
        GET /t?a=blah
    --- response_body
    blah



    === TEST 3:
    --- config
        location = /t {
            echo ok;
        }
    --- request
        GET /t?a=blah
    --- response_body
    ok

Here all the test blocks in this file have a default "--- config" section configured. Some of the test blocks can still specify its own "--- config" section to override the default, as in the `TEST 3` test block above.

You can also make the defaults applicable to all the test files. Just create a subclass of Test::Nginx::Socket (or one of its subclasses like Test::Nginx::Socket::Lua, as in,

    package t::MyTester;

    use Test::Nginx::Socket -Base;

    add_block_preprocessor(sub {
        my $block = shift;

        if (!defined $block->config) {
            $block->set_value("config", <<'_END_');
    location = /t {
        echo $arg_a;
    }
    _END_
        }
    });

    1;

Save this as file t/MyTester.pm. And then in one of your test file:

    use t::MyTester 'no_plan';

    run_tests();

    __DATA__

    === TEST 1:
    --- request
        GET /t?a=3
    --- response_body
    3



    === TEST 2:
    --- request
        GET /t?a=blah
    --- response_body
    blah

You can do the same with the "--- http_config" section, or even inventing your own new sections. This is very powerful.

add_response_body_check

Below is an example for doing HTML title checks:

    add_response_body_check(sub {
            my ($block, $body, $req_idx, $repeated_req_idx, $dry_run) = @_;

            my $name = $block->name;
            my $expected_title = $block->resp_title;

            if ($expected_title && !ref $expected_title) {
                $expected_title =~ s/^\s*|\s*$//gs;
            }

            if (defined $expected_title) {
                SKIP: {
                    skip "$name - resp_title - tests skipped due to $dry_run", 1 if $dry_run;

                    my $title;
                    if ($body =~ m{<\s*title\s*>\s*(.*?)<\s*/\s*title\s*>}) {
                        $title = $1;
                        $title =~ s/\s*$//s;
                    }

                    is_str($title, $expected_title,
                           "$name - resp_title (req $repeated_req_idx)" );
                }
            }
        });

is_str

http2

Under the hood, the test scaffold uses the `curl` command-line utility to do the wire communication with the NGINX server. The `curl` utility must be recent enough to support both the "--http2" and "--http2-prior-knowledge" command-line options.

WARNING: not all the sections and features are supported when this "--- http2" section is specified. For example, this section cannot be used with "--- pipelined_requests" or "--- raw_request".

See also the TEST_NGINX_USE_HTTP2 system environment for the "http2" test mode.

config

        location /echo {
            echo_before_body hello;
            echo world;
        }

Sometimes you simply don't want to bother copying ten times the same configuration for the ten tests you want to run against your module. One way to do this is to write a config section only for the first test in your ".t" file. All subsequent tests will re-use the same config. Please note that this depends on the order of test, so you should run "prove" with variable "TEST_NGINX_NO_SHUFFLE=1" (see below for more on this variable).

Please note that config section goes through environment variable expansion provided the variables to expand start with TEST_NGINX. So, the following is a perfectly legal (provided "TEST_NGINX_HTML_DIR" is set correctly):

    location /main {
        echo_subrequest POST /sub -f $TEST_NGINX_HTML_DIR/blah.txt;
    }

http_config

    upstream database {
        postgres_server     127.0.0.1:$TEST_NGINX_POSTGRESQL_PORT
                            dbname=ngx_test user=ngx_test
                            password=wrong_pass;
    }

As you guessed from the example above, this section goes through environment variable expansion (variables have to start with TEST_NGINX).

main_config

This section goes through environment variable expansion (variables have to start with TEST_NGINX).

post_main_config

server_name

init

request

In its most basic form, this section looks like that:

    --- request
    GET

This will just do a GET request on the root (i.e. /) of the server using HTTP/1.1.

Of course, you might want to test something else than the root of your web server and even use a different version of HTTP. This is possible:

    --- request
    GET /foo HTTP/1.0

Please note that specifying HTTP/1.0 will not prevent Test::Nginx from sending the "Host" header. Actually Test::Nginx always sends 2 headers: "Host" (with value localhost) and "Connection" (with value "close" for simple requests and keep-alive for all but the last pipelined_requests).

You can also add a content to your request:

    --- request
    POST /foo
    Hello world

Test::Nginx will automatically calculate the content length and add the corresponding header for you.

This being said, as soon as you want to POST real data, you will be interested in using the more_headers section and using the power of Test::Base filters to urlencode the content you are sending. Which gives us a slightly more realistic example:

    --- more_headers
    Content-type: application/x-www-form-urlencoded
    --- request eval
    use URI::Escape;
    "POST /rrd/foo
    value=".uri_escape("N:12345")

Sometimes a test is more than one request. Typically you want to POST some data and make sure the data has been taken into account with a GET. You can do it using arrays:

    --- request eval
    ["POST /users
    name=foo", "GET /users/foo"]

This way, REST-like interfaces are pretty easy to test.

When you develop nifty nginx modules you will eventually want to test things with buffers and "weird" network conditions. This is where you split your request into network packets:

    --- request eval
    [["POST /users\nna", "me=foo"]]

Here, Test::Nginx will first send the request line, the headers it automatically added for you and the first two letters of the body ("na" in our example) in ONE network packet. Then, it will send the next packet (here it's "me=foo"). When we talk about packets here, this is not exactly correct as there is no way to guarantee the behavior of the TCP/IP stack. What Test::Nginx can guarantee is that this will result in two calls to "syswrite".

A good way to make almost sure the two calls result in two packets is to introduce a delay (let's say 2 seconds)before sending the second packet:

    --- request eval
    [["POST /users\nna", {value => "me=foo", delay_before => 2}]]

Of course, everything can be combined till your brain starts boiling ;) :

    --- request eval
    use URI::Escape;
    my $val="value=".uri_escape("N:12346");
    [["POST /rrd/foo
    ".substr($val, 0, 6),
    {value => substr($val, 6, 5), delay_before=>5},
    substr($val, 11)],  "GET /rrd/foo"]

Adding comments before the actual request spec is also supported, for example,

   --- request
   # this request contains the URI args
   # "foo" and "bar":
   GET /api?foo=1&bar=2

request_eval

    --- request_eval
    "POST /echo_body
    hello\x00\x01\x02
    world\x03\x04\xff"

should be replaced by:

    --- request eval
    "POST /echo_body
    hello\x00\x01\x02
    world\x03\x04\xff"

pipelined_requests

Here is an example from ngx_lua's test suite:

    === TEST 7: discard body
    --- config
        location = /foo {
            content_by_lua '
                ngx.req.discard_body()
                ngx.say("body: ", ngx.var.request_body)
            ';
        }
        location = /bar {
            content_by_lua '
                ngx.req.read_body()
                ngx.say("body: ", ngx.var.request_body)
            ';
        }
    --- pipelined_requests eval
    ["POST /foo
    hello, world",
    "POST /bar
    hiya, world"]
    --- response_body eval
    ["body: nil\n",
    "body: hiya, world\n"]

more_headers

    --- more_headers
    X-Foo: blah

This will add "X-Foo: blah" to the request (on top of the automatically generated headers like "Host", "Connection" and potentially "Content-Length").

curl

For example,

    --- request
    GET /foo/bar?baz=3

    --- more_headers
    X-Foo: 3
    User-Agent: openresty

    --- curl

will produce the following line (to "stderr") while running this test block:

    # curl -i -sS -H 'X-Foo: 3' -A openresty 'http://127.0.0.1:1984/foo/bar?baz=3'

You need to remember to set the "TEST_NGINX_NO_CLEAN" environment to 1 to prevent the nginx and other processes from quitting automatically upon test exits.

response_body_filters

The filters can be specified either as names (for builtin filters) or as arbitrary Perl subroutine references.

The following builtin filter names are supported:

md5_hex

sha1_hex

length

uc

lc

ucfirst

lcfirst

Their meanings are self-explanatory.

Here is an example:

    === TEST 1:
    --- config
        location = /t {
            echo hello;
        }
    --- request
        GET /t
    --- response_body_filters
    uc
    --- response_body
    HELLO

Here the actual response body data, "hello", will go through the fitler, "uc", to become all-upper-case, before getting matched against the expected pattern specified by the "response_body" section, "HELLO".

The example above can be rewritten by using raw Perl subroutine reference values:

    === TEST 1:
    --- config
        location = /t {
            echo hello;
        }
    --- request
        GET /t
    --- response_body_filters eval
    \&CORE::uc
    --- response_body
    HELLO

To reference builtin Perl functions like "\&CORE::uc" and "\&CORE::lc", you need at least perl 5.16.

Multiple builtin filter names can be specified at the same time and they will be applied in order. For example,

    === TEST 2:
    --- config
        location = /t {
            echo hello;
        }
    --- request
        GET /hello
    --- response_body_filters
    uc lc
    --- response_body
    hello

If the response_body_filters value can also be an array reference, mostly useful for specifying multiple Perl subroutine references as the filters:

    === TEST 3:
    --- config
        location = /t {
            echo hello;
        }
    --- request
        GET /hello
    --- response_body_filters eval
    [\&CORE::uc, \&CORE::lc]
    --- response_body
    hello

If the response_body_filters value can also be an two-dimensional array reference, it means the actual response body data will be "isolatedly" applied by the indexed array's filters:

    === TEST 4:
    --- config
        location = /t {
            echo hello;
        }
    --- request eval
        ['GET /t', 'GET /t']
    --- response_body_filters eval
    [[\&CORE::uc, \&CORE::lc], [\&CORE::uc]]
    --- response_body eval
    ['hello', 'HELLO']

response_body

    --- response_body
    hello

If the test is made of multiple requests, then the response_body MUST be an array and each request MUST return the corresponding expected body:

    --- request eval
    ["GET /hello", "GET /world"]
    --- response_body eval
    ["hello", "world"]

response_body_eval

    --- response_body_eval
    "hello\x00\x01\x02
    world\x03\x04\xff"

should be replaced by:

    --- response_body eval
    "hello\x00\x01\x02
    world\x03\x04\xff"

response_body_like

    --- response_body_like
    ^elapsed 0\.00[0-5] sec\.$

If the test is made of multiple requests, then response_body_like MUST be an array and each request MUST match the corresponding pattern.

response_body_unlike

response_headers

    --- response_headers
    Content-Type: application/x-resty-dbd-stream

Of course, you can specify many headers in this section:

    --- response_headers
    X-Resty-DBD-Module:
    Content-Type: application/x-resty-dbd-stream

The test will be successful only if all headers are found in the response with the appropriate values.

If the test is made of multiple requests, then response_headers MUST be an array and each element of the array is checked against the response to the corresponding request.

response_headers_like

    --- response_headers_like
    X-Resty-DBD-Module: ngx_drizzle \d+\.\d+\.\d+
    Content-Type: application/x-resty-dbd-stream

This will check that the response's "Content-Type" is application/x-resty-dbd-stream and that the "X-Resty-DBD-Module" matches "ngx_drizzle \d+\.\d+\.\d+".

The test will be successful only if all headers are found in the response and if the values match the patterns.

If the test is made of multiple requests, then response_headers_like MUST be an array and each element of the array is checked against the response to the corresponding request.

raw_response_headers_like

    --- raw_response_headers_like: Location: http://localhost(?::\d+)?/foo\r\n

As usual, if the test is made of multiple requests, then raw_response_headers_like MUST be an array.

raw_response_headers_unlike

error_code

    --- error_code: 302

If the test is made of multiple requests, then error_code MUST be an array with the expected value for the response status of each request in the test.

error_code_like

    --- error_code_like: ^(?:500)?$

If the test is made of multiple requests, then error_code_like MUST be an array with the expected value for the response status of each request in the test.

timeout

Note that, just as almost all the timeout settings in the Nginx world, this timeout also specifies the maximum waiting time between two successive I/O events on the same socket handle, rather than the total waiting time for the current socket operation.

When the timeout setting expires, a test failure will be triggered with the message "ERROR: client socket timed out - TEST NAME", unless you have specified "--- abort" at the same time.

Here is an example:

    === TEST 1: test timeout
    --- location
        location = /t {
            echo_sleep 1;
            echo ok;
        }
    --- request
        GET /t
    --- response_body
    ok
    --- timeout: 1.5

An optional time unit can be specified, for example,

    --- timeout: 50ms

Acceptable time units are "s" (seconds) and "ms" (milliseconds). If no time unit is specified, then default to seconds.

Default to 3s.

error_log_file

Right now, it will not affect the "--- error_log" section and etc accordingly.

error_log

For example,

    === TEST 1: matched with j
    --- config
        location /re {
            content_by_lua '
                m = ngx.re.match("hello, 1234", "([0-9]+)", "j")
                if m then
                    ngx.say(m[0])
                else
                    ngx.say("not matched!")
                end
            ';
        }
    --- request
        GET /re
    --- response_body
    1234
    --- error_log: pcre JIT compiling result: 1

Then the substring "pcre JIT compiling result: 1" must appear literally in a line of error.log.

Multiple patterns are also supported, for example:

    --- error_log eval
    ["abc", qr/blah/]

then the substring "abc" must appear literally in a line of error.log, and the regex "qr/blah" must also match a line in error.log.

By default, only the part of the error logs corresponding to the current request is checked. You can make it check accumulated error logs by calling the "check_accum_error_log" Perl function before calling "run_tests" in the boilerplate Perl code above the "__DATA__" line.

access_log

Below is an example:

    === TEST 1: check access log
    --- config
        location /t {
            content_by_lua_block {
                ngx.say("hello")
            }
        }

    --- request
    GET /t
    --- response_body
    hello
    --- access_log
    GET /t

abort

shutdown

    --- shutdown: 1

will make the connection stop sending data, which is the default.

no_error_log

Here is an example:

    --- no_error_log
    [error]

This test will fail when any of the line in the error.log file contains the string "[error]".

Just like the "--- error_log" section, one can also specify multiple patterns:

    --- no_error_log eval
    ["abc", qr/blah/]

Then if any line in error.log contains the string "abc" or match the Perl regex "qr/blah/", then the test will fail.

grep_error_log

You can specify a verbatim substring being matched in the error log messages, as in

    --- grep_error_log chop
    some thing we want to see

or specify a Perl regex object to match against the error log message lines, as in

    --- grep_error_log eval
    qr/something should be: \d+/

All the matched substrings in the error log messages will be concatenated by a newline character as a whole to be compared with the value of the "--- grep_error_log_out" section.

grep_error_log_out

If the filtered output varies among the repeated requests (specified by the "repeat_each" function, then you can specify a Perl array as the value, as in

    --- grep_error_log_out eval
    ["output for req 0", "output for req 1"]

log_level

For example:

    --- log_level: debug

The default error log level can be specified in the Perl code by calling the "log_level()" function, as in

    use Test::Nginx::Socket;

    repeat_each(2);
    plan tests => repeat_each() * (3 * blocks());

    log_level('warn');

    run_tests();

    __DATA__
    ...

raw_request

    --- raw_request eval
    ["POST /rrd/taratata HTTP/1.1\r
    Host: localhost\r
    Connection: close\r
    Content-Type: application/",
    "x-www-form-urlencoded\r
    Content-Length:15\r\n\r\nvalue=N%3A12345"]

This can also be useful to tests "invalid" request lines:

    --- raw_request
    GET /foo HTTP/2.0 THE_FUTURE_IS_NOW

http09

Below is an example from ngx_headers_more module's test suite:

    === TEST 38: HTTP 0.9 (set)
    --- config
        location /foo {
            more_set_input_headers 'X-Foo: howdy';
            echo "x-foo: $http_x_foo";
        }
    --- raw_request eval
    "GET /foo\r\n"
    --- response_headers
    ! X-Foo
    --- response_body
    x-foo:
    --- http09

ignore_response

reload_fails

This only makes sense in the HUP reload testing mode.

user_files

    --- user_files
    >>> blah.txt
    Hello, world

will create a file named "blah.txt" in the html directory of the nginx server tested. The file will contain the text "Hello, world".

Multiple files are supported, for example,

    --- user_files
    >>> foo.txt
    Hello, world!
    >>> bar.txt
    Hello, heaven!

An optional last modified timestamp (in elpased seconds since Epoch) is supported, for example,

    --- user_files
    >>> blah.txt 199801171935.33
    Hello, world

It's also possible to specify a Perl data structure for the user files to be created, for example,

    --- user_files eval
    [
        [ "foo.txt" => "Hello, world!", 199801171935.33 ],
        [ "bar.txt" => "Hello, heaven!" ],
    ]

skip_eval

The format for this section is

    --- skip_eval
    <subtest-count>: <perl-code>

For example, to skip 3 subtests when the current operating system is not Linux:

    --- skip_eval
    3: $^O ne 'linux'

or equivalently,

    --- skip_eval: 3: $^O ne 'linux'

skip_nginx

The format for this section is

    --- skip_nginx
    <subtest-count>: <op> <version>

The <subtest-count> value must be a positive integer. The <op> value could be either ">", ">=", "<", or "<=". the <version> part is a valid nginx version number, like 1.0.2.

An example is

    === TEST 1: sample
    --- config
        location /t { echo hello; }
    --- request
        GET /t
    --- response_body
    --- skip_nginx
    2: < 0.8.54

That is, skipping 2 subtests in this test block for nginx versions older than 0.8.54.

This "skip_nginx" section only allows you to specify one boolean expression as the skip condition. If you want to use two boolean expressions, you should use the "skip_nginx2" section instead.

skip_nginx2

The format for this section is

    --- skip_nginx2
    <subtest-count>: <op> <version> and|or <op> <version>

For example:

    === TEST 1: sample
    --- config
        location /t { echo hello; }
    --- request
        GET /t
    --- response_body
    --- skip_nginx2
    2: < 0.8.53 and >= 0.8.41

skip_openssl

The format for this section is

    --- skip_openssl
    <subtest-count>: <op> <version>

The <subtest-count> value must be a positive integer. The <op> value could be either ">", ">=", "<", or "<=". The <version> part is a valid OpenSSL version number, like 1.1.1 or "1.1.0h".

An example is

    === TEST 1: sample
    --- config
        location /t { echo hello; }
    --- request
        GET /t
    --- response_body
    --- skip_openssl
    2: < 1.1.1

That is, skipping 2 subtests in this test block for OpenSSL versions older than 1.1.1.

This "skip_openssl" section only allows you to specify one boolean expression as the skip condition.

todo

The format for this section is

    --- todo
    <subtest-count>: <reason>

The <subtest-count> value must be a positive integer.

<reason> is logged when you run tests with --directives.

stap

Here's an example:

    === TEST 1: stap sample
    --- config
        location /t { echo hello; }
    --- stap
    probe process("nginx").function("ngx_http_finalize_request")
    {
        printf("finalize %s?%s\n", ngx_http_req_uri($r),
               ngx_http_req_args($r))
    }
    --- stap_out
    finalize /test?a=3&b=4
    --- request
    GET /test?a=3&b=4
    --- response_body
    hello

There's some macros that can be used in the "--- stap" section value. These macros will be expanded by the test scaffold automatically.

"F(function_name)"

This expands to "probe process("nginx").function("function_name")". For example, the sample above can be rewritten as

    === TEST 1: stap sample
    --- config
        location /t { echo hello; }
    --- stap
    F(ngx_http_finalize_request)
    {
        printf("finalize %s?%s\n", ngx_http_req_uri($r),
               ngx_http_req_args($r))
    }
    --- stap_out
    finalize /test?a=3&b=4
    --- request
    GET /test?a=3&b=4
    --- response_body
    hello
    

"T()"

This macro will be expanded to "println("Fire ", pp())".

"M(static-probe-name)"

This macro will be expanded to "probe process("nginx").mark("static-probe-name")".

For example,

    M(http-subrequest-start)
    {
        ...
    }
    

will be expanded to

    probe process("nginx").mark("http-subrequest-start")
    {
        ...
    }
    

stap_out

stap_out_like

stap_out_unlike

wait

udp_listen

The UDP server will first read and discard a datagram and then send back a datagram with the content specified by the "udp_reply" section value.

Here is an example:

    === TEST 1: udp access
    --- config
        location = /t {
            content_by_lua '
                local udp = ngx.socket.udp()
                udp:setpeername("127.0.0.1", 19232)
                udp:send("blah")
                local data, err = udp:receive()
                ngx.say("received: ", data)
            ';
        }
    --- udp_listen: 19232
    --- udp_reply: hello world
    --- request
    GET /t
    --- response_body
    received: hello world

Datagram UNIX domain socket is also supported if a path name ending with ".sock" is given to this directive. For instance,

    === TEST 2: datagram unix domain socket access
    --- config
        location = /t {
            content_by_lua '
                local udp = ngx.socket.udp()
                udp:setpeername("unix:a.sock")
                udp:send("blah")
                local data, err = udp:receive()
                ngx.say("received: ", data)
            ';
        }
    --- udp_listen: a.sock
    --- udp_reply: hello world
    --- request
    GET /t
    --- response_body
    received: hello world

udp_reply

You can also specify a delay time before sending out the reply via the "udp_reply_delay" section. By default, there is no delay.

An array value can be specified to make the embedded UDP server to send multiple replies as specified, for example:

    --- udp_reply eval
    [ "hello", "world" ]

This section also accepts a Perl subroutine value that can be used to generate dynamic response packet or packets based on the actual query, for example:

    --- udp_reply eval
    sub {
        my $req = shift;
        return "hello, $req";
    }

The custom Perl subroutine can also return an array reference, for example,

    --- udp_reply eval
    sub {
        my $req = shift;
        return ["hello, $req", "hiya, $req"];
    }

See the "udp_listen" section for more details.

udp_reply_delay

It is 0 delay by default.

An optional time unit can be specified, for example,

    --- udp_reply_delay: 50ms

Acceptable time units are "s" (seconds) and "ms" (milliseconds). If no time unit is specified, then default to seconds.

udp_query

For example,

    === TEST 1: udp access
    --- config
        location = /t {
            content_by_lua '
                local udp = ngx.socket.udp()
                udp:setpeername("127.0.0.1", 19232)
                udp:send("blah")
                local data, err = udp:receive()
                ngx.say("received: ", data)
            ';
        }
    --- udp_listen: 19232
    --- udp_reply: hello world
    --- request
    GET /t
    --- udp_query: hello world
    --- response_body
    received: hello world

tcp_listen

    --- tcp_listen: 12345

Stream-typed unix domain socket is also supported. Just specify the path to the socket file, as in

    --- tcp_listen: /tmp/my-socket.sock

tcp_no_close

tcp_reply_delay

tcp_reply

Like the "udp_reply" section, this section also accepts a Perl subroutine value that can be used to generate dynamic response packet or packets based on the actual query, for example:

    --- tcp_reply eval
    sub {
        my $req = shift;
        return "hello, $req";
    }

The custom Perl subroutine can also return an array reference, for example,

    --- tcp_reply eval
    sub {
        my $req = shift;
        return ["hello, $req", "hiya, $req"];
    }

tcp_query

tcp_query_len

If "tcp_query" is specified, "tcp_query_len" defaults to the length of the value of "tcp_query".

tcp_shutdown

raw_request_middle_delay

no_check_leak

must_die

Normal request and response cycle is not done. But you can still use the "error_log" section to check if there is an error message to be seen.

This is meant to test bogus configuration is noticed and given proper error message. It is normal to see stderr error message when running these tests.

Below is an example:

    === TEST 1: bad "return" directive
    --- config
        location = /t {
            return a b c;
        }
    --- request
        GET /t
    --- must_die
    --- error_log
    invalid number of arguments in "return" directive
    --- no_error_log
    [error]

This configuration ignores "TEST_NGINX_USE_VALGRIND" "TEST_NGINX_USE_STAP" or "TEST_NGINX_CHECK_LEAK" since there is no point to check other things when the nginx is expected to die right away.

This directive is handled before checking "TEST_NGINX_IGNORE_MISSING_DIRECTIVES".

server_addr_for_client

TEST_NGINX_REUSE_PORT

TEST_NGINX_USE_HTTP2

Under the hood, the test scaffold uses the `curl` command-line utility to do the wire communication with the NGINX server. The `curl` utility must be recent enough to support both the "--http2" and "--http2-prior-knowledge" command-line options.

WARNING: not all the sections and features are supported in the "http2" test mode. For example, the pipelined_requests and raw_request will still use the HTTP/1 protocols even in the "http2" test mode. Similarly, test blocks explicitly require the HTTP 1.0 protocol will still use HTTP 1.0.

One can enable HTTP/2 mode for an individual test block by specifying the http2 section, as in

    --- http2

TEST_NGINX_VERBOSE

TEST_NGINX_BENCHMARK

When specifying a positive number as the value, then this number is used for the total number of repeated requests. For example,

    export TEST_NGINX_BENCHMARK=1000

will result in 1000 repeated requests for each test block. Default to 100000.

When a second number is specified (separated from the first number by spaces), then this second number is used for the concurrency level for the benchmark. For example,

    export TEST_NGINX_BENCHMARK='1000 10'

will result in 1000 repeated requests over 10 concurrent connections for each test block. The default concurrency level is 2 (or 1 if the number of requests is 1).

The "benchmark" testing mode will also output to stderr the actual "ab" or "weighttp" command line used by the test scaffold. For example,

    weighttp -c2 -k -n2000 -H 'Host: foo.com' http://127.0.0.1:1984/t

See also the "TEST_NGINX_BENCHMARK_WARMUP" environment.

This testing mode requires the "unbuffer" command-line utility from the "expect" package.

TEST_NGINX_BENCHMARK_WARMUP

The latencies of the warm-up requests never get included in the final benchmark results.

Only meaningful in the "benchmark" testing mode.

See also the "TEST_NGINX_BENCHMARK" environment.

TEST_NGINX_CHECK_LEAK

Specifically, it starts "weighttp" (for HTTP 1.1 "GET" requests) or "ab" (for HTTP 1.0 requests) to repeatedly hitting Nginx for seconds in a sub-process, and then after about 1 second, it will start sampling the RSS value of the Nginx process by calling the "ps" utility every 20 ms. Finally, it will output all the sample point data and the line slope of the linear regression result on the 100 sample points.

One typical output for non-leaking test cases:

    t/075-logby.t .. 3/17 TEST 2: log_by_lua_file
    LeakTest: [2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176 2176 2176 2176 2176 2176 2176
     2176 2176 2176]
    LeakTest: k=0.0

and here is an example of leaking:

    TEST 5: ngx.ctx available in log_by_lua (not defined yet)
    LeakTest: [4396 4440 4476 4564 4620 4708 4752
     4788 4884 4944 4996 5032 5080 5132 5188 5236
     5348 5404 5464 5524 5596 5652 5700 5776 5828
     5912 5964 6040 6108 6108 6316 6316 6584 6672
     6672 6752 6820 6912 6912 6980 7064 7152 7152
     7240 7340 7340 7432 7508 7508 7600 7700 7700
     7792 7896 7896 7992 7992 8100 8100 8204 8296
     8296 8416 8416 8512 8512 8624 8624 8744 8744
     8848 8848 8968 8968 9084 9084 9204 9204 9324
     9324 9444 9444 9584 9584 9704 9704 9832 9832
     9864 9964 9964 10096 10096 10488 10488 10488
     10488 10488 11052 11052]
    LeakTest: k=64.1

Even very small leaks can be amplified and caught easily by this testing mode because their slopes will usually be far above 1.0.

One can configure the number of sample points via the TEST_NGINX_CHECK_LEAK_COUNT system environment, for example, to sample 1000 data points, we can set the following environment before running the test:

    export TEST_NGINX_CHECK_LEAK_COUNT=1000

For now, only "GET", "POST", "PUT", and "HEAD" requests are supported (due to the limited HTTP support in both "ab" and "weighttp"). Other methods specified in the test cases will turn to "GET" with force.

The tests in this mode will always succeed because this mode also enforces the "dry-run" mode.

Test blocks carrying the "--- no_check_leak" directive will be skipped in this testing mode.

TEST_NGINX_CHECK_LEAK_COUNT

Defaults to 100.

TEST_NGINX_USE_HUP

TEST_NGINX_LOAD_MODULES

For example, when this environment takes the value "../../work/nginx/modules/ngx_http_headers_more_filter_module.so ../../work/nginx/modules/ngx_http_lua_module.so ", then the following snippet will be generated in nginx.conf:

    load_module ../../work/nginx/modules/ngx_http_headers_more_filter_module.so;
    load_module ../../work/nginx/modules/ngx_http_lua_module.so;

This requires at least NGINX 1.9.11 to work.

TEST_NGINX_POSTPONE_OUTPUT

For example, setting TEST_NGINX_POSTPONE_OUTPUT to 1 will have the following line in nginx.conf's http {} block:

    postpone_output 1;

and it will effectively disable the write buffering in nginx's ngx_http_write_module.

TEST_NGINX_NO_CLEAN

TEST_NGINX_NO_NGINX_MANAGER

TEST_NGINX_NO_SHUFFLE

TEST_NGINX_USE_VALGRIND

Nginx is actually started with "valgrind -q $TEST_NGINX_USE_VALGRIND --gen-suppressions=all --suppressions=valgrind.suppress", the suppressions option being used only if there is actually a valgrind.suppress file.

If this environment is set to the number 1 or any other non-zero numbers, then it is equivalent to taking the value "--tool=memcheck --leak-check=full".

TEST_NGINX_USE_RR

This feature is experimental.

TEST_NGINX_USE_STAP

You can specify the stap script in the "stap" section.

Note that you need to use the "stap-nginx" script from the "nginx-dtrace" project.

TEST_NGINX_STAP_OUT

It's common to specify "TEST_NGINX_STAP_OUT=/dev/stderr" when debugging.

TEST_NGINX_BINARY

TEST_NGINX_LOG_LEVEL

TEST_NGINX_MASTER_PROCESS

TEST_NGINX_SERVER_PORT

TEST_NGINX_CLIENT_PORT

TEST_NGINX_PORT

TEST_NGINX_SLEEP

TEST_NGINX_FORCE_RESTART_ON_TEST

TEST_NGINX_SERVROOT

TEST_NGINX_IGNORE_MISSING_DIRECTIVES

TEST_NGINX_EVENT_TYPE

For example,

    $ TEST_NGINX_EVENT_TYPE=select prove -r t

TEST_NGINX_ERROR_LOG

TEST_NGINX_RANDOMIZE

Note that only test suite without external shared and writable service dependencies (like Memcached, Redis or MySQL) can run in parallel in this way, obviously.

TEST_NGINX_WORKER_USER

    export TEST_NGINX_WORKER_USER='agentzh'

Defaults to the `root` when the master is run by `root` also. Otherwise defaults to the current user.

One can also add an optional user group separated by spaces, as in

    export TEST_NGINX_WORKER_USER='agentzh wheel'

Special Variables

Variables like "TEST_NGINX_RAND_PORT_XXX" are expanded to random and unused unprivileged ports numbers to build the configuration of the server, where "XXX" is an integer from 1985 to 65535.

For instance, "TEST_NGINX_RAND_PORT_1" will be expanded to an port number, such as 1986, and "TEST_NGINX_RAND_PORT_2" will be expanded to another port number, such as 65535.

Valgrind Integration

First ensure that your valgrind executable visible in your PATH env. And then run your test suite with the "TEST_NGINX_USE_VALGRIND" env set to true:

   TEST_NGINX_USE_VALGRIND=1 prove -r t

If you see false alarms, you do have a chance to skip them by defining a ./valgrind.suppress file at the root of your module source tree, as in

<https://github.com/chaoslawful/drizzle-nginx-module/blob/master/valgrind.suppress>

This is the suppression file for ngx_drizzle. Test::Nginx will automatically use it to start nginx with valgrind memcheck if this file does exist at the expected location.

If you do see a lot of "Connection refused" errors while running the tests this way, then you probably have a slow machine (or a very busy one) that the default waiting time is not sufficient for valgrind to start. You can define the sleep time to a larger value by setting the "TEST_NGINX_SLEEP" env:

   TEST_NGINX_SLEEP=1 prove -r t

The time unit used here is "second". The default sleep setting just fits my ThinkPad ("Core2Duo T9600").

Applying the no-pool patch to your nginx core is recommended while running nginx with valgrind:

<https://github.com/shrimp/no-pool-nginx>

The nginx memory pool can prevent valgrind from spotting lots of invalid memory reads/writes as well as certain double-free errors. We did find a lot more memory issues in many of our modules when we first introduced the no-pool patch in practice ;)

There's also more advanced features in Test::Nginx that have never documented. I'd like to write more about them in the near future ;)

Etcproxy Integration

<https://github.com/openresty/mockeagain>

The default settings in etcproxy (https://github.com/chaoslawful/etcproxy) makes this small TCP proxy split the TCP packets into bytes and introduce 1 ms latency among them.

There's usually various TCP chains that we can put etcproxy into, for example

Test::Nginx <=> nginx

  $ ./etcproxy 1234 1984

Here we tell etcproxy to listen on port 1234 and to delegate all the TCP traffic to the port 1984, the default port that Test::Nginx makes nginx listen to.

And then we tell Test::Nginx to test against the port 1234, where etcproxy listens on, rather than the port 1984 that nginx directly listens on:

  $ TEST_NGINX_CLIENT_PORT=1234 prove -r t/

Then the TCP chain now looks like this:

  Test::Nginx <=> etcproxy (1234) <=> nginx (1984)

So etcproxy can effectively emulate extreme network conditions and exercise "unusual" code paths in your nginx server by your tests.

In practice, *tons* of weird bugs can be captured by this setting. Even ourselves didn't expect that this simple approach is so effective.

nginx <=> memcached

We first start the memcached server daemon on port 11211:

   memcached -p 11211 -vv

and then we another etcproxy instance to listen on port 11984 like this

   $ ./etcproxy 11984 11211

Then we tell our t/foo.t test script to connect to 11984 rather than 11211:

  # foo.t
  use Test::Nginx::Socket;
  repeat_each(1);
  plan tests => 2 * repeat_each() * blocks();
  $ENV{TEST_NGINX_MEMCACHED_PORT} ||= 11211;  # make this env take a default value
  run_tests();

  __DATA__

  === TEST 1: sanity
  --- config
  location /foo {
       set $memc_cmd set;
       set $memc_key foo;
       set $memc_value bar;
       memc_pass 127.0.0.1:$TEST_NGINX_MEMCACHED_PORT;
  }
  --- request
      GET /foo
  --- response_body_like: STORED
  --- error_code: 201

The Test::Nginx library will automatically expand the special macro $TEST_NGINX_MEMCACHED_PORT to the environment with the same name. You can define your own $TEST_NGINX_BLAH_BLAH_PORT macros as long as its prefix is "TEST_NGINX_" and all in upper case letters.

And now we can run your test script against the etcproxy port 11984:

   TEST_NGINX_MEMCACHED_PORT=11984 prove t/foo.t

Then the TCP chains look like this:

   Test::Nginx <=> nginx (1984) <=> etcproxy (11984) <=> memcached (11211)

If "TEST_NGINX_MEMCACHED_PORT" is not set, then it will take the default value 11211, which is what we want when there's no etcproxy configured:

   Test::Nginx <=> nginx (1984) <=> memcached (11211)

This approach also works for proxied mysql and postgres traffic. Please see the live test suite of ngx_drizzle and ngx_postgres for more details.

Usually we set both "TEST_NGINX_CLIENT_PORT" and "TEST_NGINX_MEMCACHED_PORT" (and etc) at the same time, effectively yielding the following chain:

   Test::Nginx <=> etcproxy (1234) <=> nginx (1984) <=> etcproxy (11984) <=> memcached (11211)

as long as you run two separate etcproxy instances in two separate terminals.

It's easy to verify if the traffic actually goes through your etcproxy server. Just check if the terminal running etcproxy emits outputs. By default, etcproxy always dump out the incoming and outgoing data to stdout/stderr.

English Mailing List

Chinese Mailing List