A prototype stores linked application processes structures. When an
application process terminates, it's removed from the list. To avoid double
removal, the pointer to the next element should be set to NULL.
The issue was introduced in c8790d2a89bb.
Port's "data" field may be used by application and thus need to be set to NULL.
The issue was introduced in the f8a0992944df commit.
Found by Coverity (CID 374352).
The reference to the access_log structure is stored in the current
nxt_router_conf_t and the global nxt_router_t. When the reference is copied,
the reference counter should be adjusted accordingly.
This closes#593 issue on GitHub.
After the c8790d2a89bb commit, the SIGCHLD handler may return before processing
all awaiting PIDs. To avoid zombie processes and ensure successful main
process termination, waitpid() must be called until an error is returned.
This closes#600 issue on GitHub.
Splitting the process type connectivity matrix to 'keep ports' and 'send
ports'; the 'keep ports' matrix is used to clean up unnecessary ports after
forking a new process, and the 'send ports' matrix determines which process
types expect to get created process ports.
Unfortunately, the original single connectivity matrix no longer works because
of an application stop delay caused by prototypes. Existing applications
should not get the new router port at the moment.
Application process started with shared port (and queue) already configured.
But still waits for PORT_ACK message from router to start request processing
(so-called "ready state").
Waiting for router confirmation is necessary. Otherwise, the application may
produce response and send it to router before the router have the information
about the application process. This is a subject of further optimizations.
Request procesing loop should be started in ready handler to avoid race
between go-routine start and context ready flag assignment.
The issue introduced in 43553aa72111.
port->app field is not thread safe and should be used in main thread only.
To release port after request processing, application reference should be
obtained from corresponding request descriptor.
Because the configuration values were read from the listener's configuration,
an established WebSocket connection was unable to work properly (i. e. stuck)
if the listener was removed. The correct source of configuration values is the
request config joint.
This is related to issue #581 on GitHub.
When processing a restart request, the router sends a QUIT message to all
existing processes of the application. Then, a new shared application port is
created to ensure that new requests won't be handled by the old processes of
the application.
Unit's ASGI implementation creates a new event loop to run an application for
each thread since 542b5b8c0647. This may cause unexpected exceptions or
strange bugs if asyncio synchronisation primitives are initialised before the
application starts (e.g. globally).
Although the approach with a new event loop for the main thread is consistent
and helps to prepare the application to run in multiple threads, it can be a
source of pain for people who just want to run single-threaded ASGI
applications in Unit.
This is related to #560 issue on GitHub.
An ASGI application can cancel the Future object returned by the receive()
call. In this case, Unit's ASGI implementation should not call set_result()
because the Future is already handled. In particular, the Starlette framework
was noted to cancel the received Future.
This patch adds a done() check for the Future before attempting a set_result().
This is related to #564 issue on GitHub.
The receive() call never blocks for a GET request and always returns the same
empty body message. The Starlette framework creates a separate task when
receive() is called in a loop until an 'http.disconnect' message is received.
The 'http.disconnect' message was previously issued after the response header
had been sent. However, the correct behavior is to respond with
'http.disconnect' after sending the response is complete.
This closes#564 issue on GitHub.
A new application thread port message can be processed in the router after the
application is removed from the router. Assertion for this case is replaced by
a condition to store the new thread port until receiving the stop notification
from the application process.
In rare cases, when the destination process had finished running but no
notification of this was received yet, send could fail with an error, and the
send message structure with file descriptors could leak.
The leakage was periodically reproduced by respawn tests on FreeBSD 12.
A race may occur between the router process restart and the main process
sending a notification to the running controller. For example, a test script
detects the new process and starts performing a smoke test, but the controller
has not yet received the 'remove PID' notification, so the connection to the
router is broken and any attempt to update the configuration will cause an
error.
The solution is to perform several attempts to reconfigure Unit with a short
delay between failures.
Because of the incorrect 'last' field assignment, multiple listeners with
a TLS certificate did not initialize properly, which caused a router crash
while establishing a connection.
Test with multiple TLS listeners added.
The issue was introduced in the c548e46fe516 commit.
This closes#561 issue on GitHub.
The patch removes the "files" section from package.json to avoid future issues
with missing files. For package testing purposes, 'npm pack' is used instead
of plain 'tar' to simulate packaging more accurately.
This patch fixes a possible race between the nxt_router_conf_wait() and
nxt_router_listen_socket_release() function calls and improves the 7f1b2eaa2d58
commit fix.
In Go 1.16, the module-aware mode is enabled by default; to fall back to
previous behavior, the GO111MODULE environment variable should be set to
'auto'.
Details: https://golang.org/doc/go1.16
Listen socket is actually closed in the instant timer handler. This patch moves
the "configuration has been applied" notification to the timer handler to avoid
a situation when the user gets the response from the controller, but the listen
socket is still open in the router.
Introducing "unit.log.Log" class for "unit.log" file management.
Moving "findall()" function into TestApplicationProto.
Using "os.kill()" to send signals.
Globally installed modules require a globally installed libunit.
The "binding_pub.gyp" file is the correct version, otherwise linked module
may have unresolved symbols because libunit is not linked.
This fixes memory and shm file descriptor leakage that occurred when a large
request body was passed via shared memory. The leakage was caught with the
"test_settings_body_buffer_size" test. The main condition is the
"body_buffer_size" value exceeding 10 Mb (a shm segment). Thus, the router was
forced to split the body into several shm segments, but these buffers were not
freed because of dummy completion handlers.
Test case: "send a text message split into two fragments, then a continuation
frame with FIN = false where there is nothing to continue, then an unfragmented
text message, all sent in one chop".
The test case investigates immediate connection closing since there is no
message to continue.
The mirror server may send a response for the first frame before the test
сontinuation frame is received by the router. In this case, the test will
receive a text frame before the close frame.
This is a workaround for an issue in OpenSSL 1.1.1, where the /dev/random and
/dev/urandom files remain open after all listening sockets were removed:
- https://github.com/openssl/openssl/issues/7419
This fixes file descriptor leakage in router. Shared memory file used to
send data from router to application. These files are shared among all
processes of same application and router keeps the opened file descriptor since
06017e6e3a5f commit.
pthread_t on Solaris is an integer type with size not equal to pointer size.
To avoid warnings, type casts to and from pointer needs to be done via
uintptr_t type.
This change originally proposed by Juraj Lutter <juraj@lutter.sk>.
Instead of PTHREAD_STACK_MIN define, NetBSD requires to get minimum stack
size using sysctl(_SC_THREAD_STACK_MIN).
This change originally proposed by Juraj Lutter <juraj@lutter.sk>.
Shared app queue takes more memory than port memory. To unmap all memory pages
correct size need to be specified for munmap() call. Otherwise 4 Mb memory
leaked on each configured application removal.
The issue was introduced in 1d84b9e4b459.
After WebSocket processing, the application port was released with incorrect
reason ("got request"), unnecessarily decrementing the active request counter.
The assertion was triggered only on application removal; a test was added
for this case.
The controller process awaits the response from the router for every
configration change request. This patch adds error reporting for various error
conditions which may happen because of file descriptors or memory shortage.
Lack of a response lead to the controller awaiting the response, thus being
unable to process other client reconfiguration requests that also became stuck.
Each application in router process required fd for a request queue shared
memory. When the number of file descripts close to the limit, and port sockets
successfully opened, router needs to properly handle the errors.
This patch closes port sockets before destroying port structure to avoid
file descriptors leakage and assertion in debug build.
This partially reverts the optimisation introduced in 1d84b9e4b459 to avoid an
unpredictable block in nxt_unit_process_port_msg(). Under high load, this
function may never return control to its caller, and the external event loop
(in Node.js and Python asyncio) won't be able to process other scheduled
events.
To reproduce the issue, two request processing types are needed: 'fast' and
'furious'. The 'fast' one simply returns a small response, while the 'furious'
schedules asynchronous calls to external resources. Thus, if Unit is subjected
to a large amount of 'fast' requests, the 'furious' request processing freezes
until the high load ends.
The issue was found by Wu Jian Ping (@wujjpp) during Node.js stream
implementation discussion and relates to PR #502 on GitHub.
ServerRequest now inherit stream Readable object. ServerResponse
provides 'writable' property.
Thanks to Wu Jian Ping (@wujjpp).
This closes#274, closes#317 issues and closes#502 PR on GitHub.
When a static file larger than NXT_HTTP_STATIC_BUF_SIZE (128K) is served, two
buffers are allocated and chained; each retains the whole request memory pool.
Starting from 41331471eee7, the completion handler was called once for a linked
buffer chain, but the second buffer got lost.
This patch improves the completion handler's treatment of static buffers to
handle all linked buffers.
The nxt_unit_ctx_port_recv() function may return the NXT_UNIT_AGAIN code, in
which case an attempt to reread the message should be made.
The issue was reproduced in load testing with response sizes 16k and up.
In the rare case of a NXT_UNIT_AGAIN result, a buffer of size -1 was processed,
which triggered a 'message too small' alert; after that, the app process was
terminated.
Under high load, a queue synchonization issue may occur, starting from the
steady state when an app queue message is dequeued immediately after it has been
enqueued. In this state, the router always puts the first message in the queue
and is forced to notify the app about a new message in an empty queue using a
socket pair. On the other hand, the application dequeues and processes the
message without reading the notification from the socket, so the socket buffer
overflows with notifications.
The issue was reproduced during Unit load tests. After a socket buffer
overflow, the router is unable to notify the app about a new first message.
When another message is enqueued, a notification is not required, so the queue
grows without being read by the app. As a result, request processing stops.
This patch changes the notification algorithm by counting the notifications in
the pipe instead of getting the number of messages in the queue.
Multithreaded application may create different shared memory segments in
different threads. The segments then passed to different router threads.
Because of this multithreading, the order of adding incoming segments is
not determined and there can be situation when some of the incoming segments
are not initialized yet.
This patch simply adds check for NULL to skip non-initialized segments.
Crash reproduced during load tests with high number of simultaneous
connections (1024 and more).
Ruby threads need to be created with GVL; otherwise, an attempt to access
locked resources may occur, causing a crash.
The issue was occasionally reproduced on Ubuntu 18.04 with Ruby 2.5.1
while running test_ruby_application_threads.
This fixes a crash on exit of Node.js application. The crash reproduced
on Ubuntu 20.10 with Node.js v15.1.0. Tests 'test_node_websockets_two_clients'
and 'test_node_websockets_7_13_1__7_13_2'.
The reason of the crash is using request struct which was already freed.
The issue was introduced in 5be509fda29e.
If shared queue is empty, allocated read buffer should be explicitly
released.
Found by Coverity (CID 363943).
The issue was introduced in f5ba5973a0a3.
The issue occurred under highly concurrent request load in Go applications.
Such applications are multi-threaded but use a single libunit context; any
thread-safe code in the libunit context is only required for Go applications.
As a result of improper request state reset, the recycled request structure was
recovered in the released state, so further operations with this request
resulted in 'response already sent' warnings. However, the actual response was
never delivered to the router and the client.
The issue only occurred in Go applications because "port_send" is overloaded
only in Go. To reproduce it, send multiple concurrent requests to the
application after it has initialised. The warning message "[unit] [go] port
NNN:dd not found" is the first visible aspect of the issue; the second and more
valuable one is a closed connection, an error response, or a hanging response to
some requests.
When the application starts, it is unaware of the router's worker thread ports,
so it requests the ports from the router after receiving requests from the
corresponding router worker threads. When multiple requests are processed
simultaneously, the router port may be required by several requests, so request
processing starts only after the application receives the required port
information. The port should be added to the Go port repository after its
'ready' flag is updated. Otherwise, Unit may start processing some requests and
use the port before it is in the repository.
The issue was introduced in changeset 78836321a126.
Application shared queue only capable to pass one shared memory buffer.
The rest buffers in chain needs to be send directly to application in response
to REQ_HEADERS_AC message.
The issue can be reproduced for configurations where 'body_buffer_size' is
greater than memory segment size (10 Mb). Requests with body size greater
than 10 Mb are just `stuck` e.g. not passed to application awaiting for more
data from router.
The bug was introduced in 1d84b9e4b459 (v1.19.0).
ASGI threads read all the requests from the queue before start processing it.
This why test need to wait a little to let the ASGI thread start request
processing and block. In virtual environment any thread or process may
be delayed and only method to avoid racing is a reasonable sleep increase.
The PORT_ACK message is the router's response to the application's NEW_PORT
message. After receiving PORT_ACK, the application is safe to process requests
using this port.
This message avoids a racing condition when the application starts processing a
request from the shared queue and sends REQ_HEADERS_ACK. The REQ_HEADERS_ACK
message contains the application port ID as reply_port, which the router uses
to send request data. When the application creates a new port, it
immediately sends it to the main router thread. Because the request is
processed outside the main thread, a racing condition can occur between the
receipt of the new port in the main thread and the receipt of REQ_HEADERS_ACK
in the worker router thread where the same port is specified as reply_port.
Max Romanov