Nginx

Build and run

git clone https://github.com/nginx/nginx
cd nginx
./auto/configure --prefix=/code/build/nginx && make && make install
gdb /code/build/nginx/sbin/nginx

By default, Nginx will start in daemon mode, i.e., it will fork a new process and the parent process just dies trivially, so we need to set follow-fork-mode child in GDB to follow the child process.

Worker

call graph

ngx_worker_process_cycle
    -> ngx_process_events_and_timers
        -> ngx_process_events, i.e., ngx_epoll_process_events

see https://nginx.org/en/docs/dev/development_guide.html#event_loop

There are multiple worker processes listing to the same port number, so when a new connect() request comes from client, all worker processes are woken up to accept() this socket connection. This is called “shock group” problem. Newer Linux kernel already solve this problem, but Nginx solves this problem in application layer nevertheless. See code below

/* ngx_event.c#ngx_process_events_and_timers */

    if (ngx_use_accept_mutex) {
        if (ngx_accept_disabled > 0) {
            ngx_accept_disabled--;

        } else {
            if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
                return;
            }

            if (ngx_accept_mutex_held) {
                flags |= NGX_POST_EVENTS;

            } else {
                if (timer == NGX_TIMER_INFINITE
                    || timer > ngx_accept_mutex_delay)
                {
                    timer = ngx_accept_mutex_delay;
                }
            }
        }
    }

Worker process holds accept mutex before polling events, such that only one process is woken up to accept the new connection. Note that it changes flags: flags |= NGX_POST_EVENTS;. Also, it should process the socket event as fast as possible. Otherwise, newer connect() events cannot be polled timely. Below is async trick of Nginx!

/* ngx_epoll_module.c */

            if (flags & NGX_POST_EVENTS) {
                queue = rev->accept ? &ngx_posted_accept_events
                                    : &ngx_posted_events;

                ngx_post_event(rev, queue);

            } else {
                rev->handler(rev);
            }

The above code shows that once NGX_POST_EVENTS flag is set, the event will be put to posted queue instead of being executed immediately. So the accept mutex can be quickly released.

How configuration works

Nginx has its own config parser.

ngx_conf_parse
    -> ngx_conf_handler

Inside ngx_conf_handler function, it will scan all valid configs and when a match is found, it calls the cmd set method rv = cmd->set(cf, cmd, conf);. See ngx_core_commands for example set functions.

stats endpoints

Nginx provides an http endpoint to report stats. In a box running nginx, type curl localhost/nginx_status, you will get blow metrics.

Active connections: 11
server accepts handled requests
       2058893 2058893 11368596
Reading: 0 Writing: 1 Waiting: 10

TODO

Answer below questions:

1. Why Nginx can solve C10K problem? Why it is fast?
2. Can Nginx substitute HAproxy?
3. Why people choose Nginx over other choices?

Action:

1. Build a website with this setup: Nginx + gunicorn + Flask

Reference

• https://nginx.org/en/docs/