This issue was introduced in libunit commit (e0f0cd7d244a). All port
sockets in application should be in blocking mode whereas Unit itself
operates non-blocking sockets.
Having non-blocking sockets in application may cause send error during
intensive response packets generation.
See https://mailman.nginx.org/pipermail/unit/2018-October/000080.html.
Router requests application start by sending port message with application
parameters to master. In response router gets application port. Stream id
used to link request and response. If application process terminates
unexpectedly (e.g. before ready state), it is required to notify router about
application start failure. For this purpose stream id is stored in process
initialization parameters.
Stream id sequentially incremented and starts from 1 in case of router
resurrection. New router application start stream id may coincide with
stream id of currently running/terminating application processes. In such
cases router may receive REMOVE_PID message with same stream id as used in
application start request and mistakenly reports application start failure.
This commit tries to avoid such errors by resetting stream for processes in
ready state.
Router reconfiguration may lead to starting new application processes, opening
listen ports etc. These actions are asynchronous and require message
exchange with master process. Router stores reference for controller
(response) port in structure associated with reconfiguration process (tmcf).
The port used to provide reconfiguration response (either success or failed)
to controller.
This commit increases response port use counter to make sure port structure
not freed while router waits for result of asynchronous operations.
Attribute 'application' is optional in listener object. This commit adds
missing code to reset 'lscf' to avoid garbage and crash when the attribute
is absent.
Unit master process restarts the router if the router accidentally dies.
New router process receives the configuration from controller and starts
configured applications. The information of running applications cannot
be transferred to router because currently there is no persistent application
identifier. To avoid orphan application processes started by died router,
master process stops all currently running applications once it receives
SIGCHLD for router process.
For accurate app descriptor release, it is required to count the number of
use counts. Use count increased when:
- app linked to configuration app queue;
- socket conf stores pointer to app;
- request for start app process posted to router service thread;
Application port has pointer to app, but it does not increase use count
to avoid use count loop.
Timer needs a pointer to nxt_timer_t which is stored in engine timers tree.
nxt_timer_t now resides in nxt_app_joint_t and does not lock the application.
Start process port RPC handlers is also linked to nxt_app_joint_t.
App joint (nxt_app_joint_t) is a 'weak pointer':
- single threaded;
- use countable;
- store pointer to nxt_app_t (which can be NULL);
nxt_app_t has pointer to nxt_app_joint_t and update its pointer to app.
Unit router process may send mmap file decritptor to the application process
for further information exchange. During this process there may be various
errors, which should be described in application error log. If file descriptor
cannot be properly transferred with 'new mmap' message, fd variable will
be assigned to -1 and further syscalls using this file descriptor will fail.
For 'new port' message fd is checked in the same way.
This commit adds early 'invalid file descriptor' diagnostic and write
corresponding message to error log.
Found by Coverity (CID 308515).
For the optimization purpose, function nxt_unit_remove_process() expects
lib->mutex to be locked. The function then moves ports queue into temporary
queue and releases mutex. In nxt_unit_done() there were two errors: mutex was
not locked before nxt_unit_remove_process() call and mutex was not destroyed.
It is hard to tell what was possible negative impact of this errors.
Found by Coverity (CID 308517).
Changeset #699 fixes shared memory allocation: continous buffer with
requested size should be allocated or function failed. For body longer
than 10 Mb, this allocation will definitely fails.
For body buffer it is not required to send it in a single continous buffer,
so, need to request minimum reasonable amount of shared memory and try to
extend it, if possible or allocate next buffer.
Previously, one shared memory chunk was allocated under mutex and other
chunks (if required) were allocated using atomic operations. So such
allocation is not guaranteed and the result buffer can be less than
requested.
This commit moves multiple chunks allocation under mutex and guarantees
the result buffer is large enough.
This retains application during reconfiguration if an incoming connection
was already established before the reconfiguration, but no request was yet
created for the connection and thus no application was yet assigned to
the request.
If the first buffer contained only header and there were other buffers in chain,
those buffers were not processed. In particular, this broke Perl application
responses with empty body.
- Pre-fork 'processes.spare' application processes;
- fork more processes to keep 'processes.spare' idle processes;
- fork on-demand up to 'processes.max' count;
- scale down idle application processes above 'processes.spare' after
'processes.idle_timeout';
- number of concurrently started application processes also limited by
'processes.spare' (or 1, if spare is 0).
New optional configuration parameter introduced: limits.reschedule_timeout.
Default value 1 second. In the case when request is written to the port
socket 'in advance', it is called 'pending'.
On every completed request, the head of pending request is checked against
reschedule timeout. If this request waiting for execution longer than
timeout, it is cancelled, new port selected for this request.
Application timeout limits maximum time of worker response in processing
particular request. Not including the time required to start worker,
time in request queue etc.
This patch increase precedence of non-started worker over busy worker.
1. idle worker;
2. start new worker;
3. busy worker, but can accept request in advance;
To communicate with the Go program, Unit setup environment variable
named NXT_GO_PORTS with value contains Unit version, stream id to confirm
application is started, and Unit ports information. Go Unit package parses
this string and compares runtime version with compile time version. In case
of parse error or version mismatch, ListenAndServe() returns with the error.
Passing unsafe.Pointers (void *) from Go to C is complicated by an attempt
to make such pointers less unsafe.
A straightforward optimization is to replace 'unsafe.Pointer' with 'uintptr'
(thanks to Xin Huang for the idea: https://stackoverflow.com/a/44826533 ).
As a result, request registry with mutex is gone.
Go request registration should be removed before C request memory freed.
C request address used as a key in Go map. Freed memory can be instantly
reused for other request and older request registration should removed at this
point to avoid collisions.
- Main process should be connected to all other processes.
- Controller should be connected to Router.
- Router should be connected to Controller and all Workers.
- Workers should be connected to Router worker thread ports only.
This filtering helps to avoid unnecessary communication and various errors
during massive application workers stop / restart.
"All problems in computer science can be
solved by another level of indirection"
Butler Lampson
Completion handlers for application response buffers executed after
sending the data to client. Application worker can be stopped right
after send response buffers to router. Worker stop causes removal
of all data structures for the worker.
To prevent shared memory segment unmap, need to count the number of
buffers which uses it. So instead of direct reference to shared
memory, need to reference to intermediate 'handler' structure with
use counter and pointer to shared memory.
Two different router threads may send different requests to single
application worker. In this case shared memory fds from worker
to router will be send over 2 different router ports. These fds
will be received and processed by different threads in any order.
This patch made possible to add incoming shared memory segments in
arbitrary order. Additionally, array and memory pool are no longer
used to store segments because of pool's single threaded nature.
Custom array-like structure nxt_port_mmaps_t introduced.
This allows to use shared memory to communicate with main process.
This patch changes shared memory segment format and breaks compatibility
with older modules.
- Each sendmsg() transmits no more than port->max_size payload data.
- Longer buffers are fragmented and send using multiple sendmsg() calls.
- On receive side, buffers are connected in chain.
- Number of handler calls is the same as number of nxt_port_socket_write()
calls.
- nxt_buf_make_plain() function introduced to make single plain buffer from
the chain.
Only purpose of request<->app link instance is to be enqueued in application
requests queue.
It is possible to avoid request<->app link allocation from memory pool in
case when spare application port is available. Instance from local stack
can be used to prepare and send message to application.
Port message handler may perform fork() and then close port read file
descriptor and enable write on same event fd. Next read attempt in this case
may cause different errors in log file.
Remove pid proxying to worker engines implementation was originally
overcomplicated. Memory pool and 2 engine posts (there and back again) are
optimized out and replaced with band new nxt_port_post() call.
Request <-> application link structure (nxt_req_app_link_t) used to register
the request in application request queue (nxt_app_t.requests) and generate
application-specific port message.
Now it is allocated from request pool. This pool created for request parsing
and used to allocate and store information specific to this request.
Request can be processed in thread different from the thread where the
connection originally handled.
Because of possible racing conditions, using original connection structures
is unsafe. To solve this, error condition is registered in 'ra' (request <->
application link) and traversed back to original connection thread where
the error message can be generated and send back to client.
For empty write queue cases, it is possible to avoid allocation and enqueue
send message structures. Send message initialized on stack and passed to
write handler. If immediate write fails, send message allocated from engine
pool and enqueued.
Use counter helps to simplify logic around port and application free.
Port 'post' function introduced to simplify post execution of particular
function to original port engine's thread.
Write message queue is protected by mutex which makes port write operation
thread safe.
To allow use port from different threads, the first step is to avoid using
port's memory pool for temporary allocations required to send data through
the port. Including but not limited by:
- buffers for data;
- send message structures;
- new mmap fd notifications;
It is still safe to use port memory pool for incoming buffers allocations
because recieve operation bound to single thread.
Worker threads ports need to receive 'remove pid' message to properly handle
application process exit case and finish requests processed by particular
application worker. Main process send 'remove pid' notification to service
thread port only and this message must be 'proxied' to other running engines.
Separate memory pool created for this message. For each engine structure
required to post message to engine allocate from the pool using 'retain'
allocation method. After successfull post structure will be freed using
'release' method. To completely destroy poll one more 'release' should be
called to release initial reference count.
I'm afraid this should be simplified using good old malloc() and free() calls.
Do not reuse shared memory segment with different port until this segment
successfully received and indexed on other side. However, segment can be used
to transfer data via the port it was sent at any time.
PHP SAPI tries to read body for POST request before registering
header-specific variables. For other methods, read_post_body() called by SAPI
after variables registration.
This closes#10 issue on GitHub.
Application free ports is a queue (double linked list) protected with mutex.
After successfull request parsing, each router thread (1) tries to get port
from this list. If this list is empty, (2) start worker request posted to main
router thread. Another thread may release port between (1) and (2).
This fix adds an attempt to get port from free ports list at the beginning of
start worker action in main thread.
Max Romanov