This fixes file descriptor leakage in router. Shared memory file used to
send data from router to application. These files are shared among all
processes of same application and router keeps the opened file descriptor since
06017e6e3a5f commit.
pthread_t on Solaris is an integer type with size not equal to pointer size.
To avoid warnings, type casts to and from pointer needs to be done via
uintptr_t type.
This change originally proposed by Juraj Lutter <juraj@lutter.sk>.
Instead of PTHREAD_STACK_MIN define, NetBSD requires to get minimum stack
size using sysctl(_SC_THREAD_STACK_MIN).
This change originally proposed by Juraj Lutter <juraj@lutter.sk>.
Shared app queue takes more memory than port memory. To unmap all memory pages
correct size need to be specified for munmap() call. Otherwise 4 Mb memory
leaked on each configured application removal.
The issue was introduced in 1d84b9e4b459.
After WebSocket processing, the application port was released with incorrect
reason ("got request"), unnecessarily decrementing the active request counter.
The assertion was triggered only on application removal; a test was added
for this case.
The controller process awaits the response from the router for every
configration change request. This patch adds error reporting for various error
conditions which may happen because of file descriptors or memory shortage.
Lack of a response lead to the controller awaiting the response, thus being
unable to process other client reconfiguration requests that also became stuck.
Each application in router process required fd for a request queue shared
memory. When the number of file descripts close to the limit, and port sockets
successfully opened, router needs to properly handle the errors.
This patch closes port sockets before destroying port structure to avoid
file descriptors leakage and assertion in debug build.
This partially reverts the optimisation introduced in 1d84b9e4b459 to avoid an
unpredictable block in nxt_unit_process_port_msg(). Under high load, this
function may never return control to its caller, and the external event loop
(in Node.js and Python asyncio) won't be able to process other scheduled
events.
To reproduce the issue, two request processing types are needed: 'fast' and
'furious'. The 'fast' one simply returns a small response, while the 'furious'
schedules asynchronous calls to external resources. Thus, if Unit is subjected
to a large amount of 'fast' requests, the 'furious' request processing freezes
until the high load ends.
The issue was found by Wu Jian Ping (@wujjpp) during Node.js stream
implementation discussion and relates to PR #502 on GitHub.
ServerRequest now inherit stream Readable object. ServerResponse
provides 'writable' property.
Thanks to Wu Jian Ping (@wujjpp).
This closes#274, closes#317 issues and closes#502 PR on GitHub.
When a static file larger than NXT_HTTP_STATIC_BUF_SIZE (128K) is served, two
buffers are allocated and chained; each retains the whole request memory pool.
Starting from 41331471eee7, the completion handler was called once for a linked
buffer chain, but the second buffer got lost.
This patch improves the completion handler's treatment of static buffers to
handle all linked buffers.
The nxt_unit_ctx_port_recv() function may return the NXT_UNIT_AGAIN code, in
which case an attempt to reread the message should be made.
The issue was reproduced in load testing with response sizes 16k and up.
In the rare case of a NXT_UNIT_AGAIN result, a buffer of size -1 was processed,
which triggered a 'message too small' alert; after that, the app process was
terminated.
Under high load, a queue synchonization issue may occur, starting from the
steady state when an app queue message is dequeued immediately after it has been
enqueued. In this state, the router always puts the first message in the queue
and is forced to notify the app about a new message in an empty queue using a
socket pair. On the other hand, the application dequeues and processes the
message without reading the notification from the socket, so the socket buffer
overflows with notifications.
The issue was reproduced during Unit load tests. After a socket buffer
overflow, the router is unable to notify the app about a new first message.
When another message is enqueued, a notification is not required, so the queue
grows without being read by the app. As a result, request processing stops.
This patch changes the notification algorithm by counting the notifications in
the pipe instead of getting the number of messages in the queue.
Multithreaded application may create different shared memory segments in
different threads. The segments then passed to different router threads.
Because of this multithreading, the order of adding incoming segments is
not determined and there can be situation when some of the incoming segments
are not initialized yet.
This patch simply adds check for NULL to skip non-initialized segments.
Crash reproduced during load tests with high number of simultaneous
connections (1024 and more).
Ruby threads need to be created with GVL; otherwise, an attempt to access
locked resources may occur, causing a crash.
The issue was occasionally reproduced on Ubuntu 18.04 with Ruby 2.5.1
while running test_ruby_application_threads.
This fixes a crash on exit of Node.js application. The crash reproduced
on Ubuntu 20.10 with Node.js v15.1.0. Tests 'test_node_websockets_two_clients'
and 'test_node_websockets_7_13_1__7_13_2'.
The reason of the crash is using request struct which was already freed.
The issue was introduced in 5be509fda29e.
If shared queue is empty, allocated read buffer should be explicitly
released.
Found by Coverity (CID 363943).
The issue was introduced in f5ba5973a0a3.
The issue occurred under highly concurrent request load in Go applications.
Such applications are multi-threaded but use a single libunit context; any
thread-safe code in the libunit context is only required for Go applications.
As a result of improper request state reset, the recycled request structure was
recovered in the released state, so further operations with this request
resulted in 'response already sent' warnings. However, the actual response was
never delivered to the router and the client.
The issue only occurred in Go applications because "port_send" is overloaded
only in Go. To reproduce it, send multiple concurrent requests to the
application after it has initialised. The warning message "[unit] [go] port
NNN:dd not found" is the first visible aspect of the issue; the second and more
valuable one is a closed connection, an error response, or a hanging response to
some requests.
When the application starts, it is unaware of the router's worker thread ports,
so it requests the ports from the router after receiving requests from the
corresponding router worker threads. When multiple requests are processed
simultaneously, the router port may be required by several requests, so request
processing starts only after the application receives the required port
information. The port should be added to the Go port repository after its
'ready' flag is updated. Otherwise, Unit may start processing some requests and
use the port before it is in the repository.
The issue was introduced in changeset 78836321a126.
Application shared queue only capable to pass one shared memory buffer.
The rest buffers in chain needs to be send directly to application in response
to REQ_HEADERS_AC message.
The issue can be reproduced for configurations where 'body_buffer_size' is
greater than memory segment size (10 Mb). Requests with body size greater
than 10 Mb are just `stuck` e.g. not passed to application awaiting for more
data from router.
The bug was introduced in 1d84b9e4b459 (v1.19.0).
ASGI threads read all the requests from the queue before start processing it.
This why test need to wait a little to let the ASGI thread start request
processing and block. In virtual environment any thread or process may
be delayed and only method to avoid racing is a reasonable sleep increase.
The PORT_ACK message is the router's response to the application's NEW_PORT
message. After receiving PORT_ACK, the application is safe to process requests
using this port.
This message avoids a racing condition when the application starts processing a
request from the shared queue and sends REQ_HEADERS_ACK. The REQ_HEADERS_ACK
message contains the application port ID as reply_port, which the router uses
to send request data. When the application creates a new port, it
immediately sends it to the main router thread. Because the request is
processed outside the main thread, a racing condition can occur between the
receipt of the new port in the main thread and the receipt of REQ_HEADERS_ACK
in the worker router thread where the same port is specified as reply_port.
This fixes the router's crash on buildbot; the reason was an unexpected 'last'
response from the application to the router arriving before the response
headers. The last buffer is not a memory buffer, so the result of accessing
memory fields is unpredictable.
The unexpected 'last' message was caused by an error in libunit; fixed in
fee8fd855a00.
Data in the queue and the socket are transmitted independently; special
READ_QUEUE and READ_SOCKET message types are used for synchronization.
The warning was accidentally committed with changeset 1d84b9e4b459.
The connection's local socket address is allocated from the connection
pool before the request is passed to the application; however, with keep-alive
connections, this field was unconditionally reset by a socket configuration
value that could be NULL. For the next request, the address was allocated
again from the same connection pool. Nonetheless, all leaked addresses
were released when the connection was closed.
The issue introduced in changeset 5c7dd85fabd5.
Large-bodied requests are added to the request hash to be found when the body
arrives. However, changeset 1d84b9e4b459 introduced a bug: the 'in_hash' flag,
used to remove the request from the hash at request release, was cleared after
the first successful request lookup. As a result, the entry was never removed.
This change aids heap usage analysis in applications.
The alloc and free functions are also required for lvlhash due to the upcoming
threading support, because using main nxt_memalign() and nxt_free() isn't safe
in a multithreaded app environment. The reason is that these functions may use
thread-local structures which aren't initialized properly in applications.
The issue was introduced in changeset 1d84b9e4b459. The request buffer was
transferred via the shared application queue, but the buffer position and the
'sent' flag were not updated after the buffer had been sent.
The Apache HTTP server benchmarking tool, ab, issues HTTP/1.0 requests with
the 'Connection: Keep-Alive' header and expects a 'Connection: Keep-Alive'
header in the response.
This is required for more flexible Python version check since ASGI works
for Python 3.5+. Version check via 'startswith()' function removed as
not consistent.
Now it is possible to specify the name of the application callable using
optional parameter 'callable'. Default value is 'application'.
This closes#290 issue on GitHub.
Buffer for application prefork request allocated from temp conf mem_pool.
If error response from main process received before buffer completion handler,
temp conf mem_pool destroyed and router may crash in completion handler.
Assertion "src/nxt_buf.c:208 assertion failed: data == b->parent" triggered
when NXT_DEBUG_ALLOC enabled in configure.
This patch disables completion handler and memory allocated for buffer
released with memory pool.
The default libunit behavior relies on blocking the recv() call for port file
descriptors, which an application may override if needed. For external
applications, port file descriptors were toggled to blocking mode before the
exec() call. If the exec() call failed, descriptor remained blocked, so the
process hanged while trying to read from it.
This patch moves file descriptor mode switch inside libunit.
Max Romanov