If shared queue is empty, allocated read buffer should be explicitly
released.
Found by Coverity (CID 363943).
The issue was introduced in f5ba5973a0a3.
The issue occurred under highly concurrent request load in Go applications.
Such applications are multi-threaded but use a single libunit context; any
thread-safe code in the libunit context is only required for Go applications.
As a result of improper request state reset, the recycled request structure was
recovered in the released state, so further operations with this request
resulted in 'response already sent' warnings. However, the actual response was
never delivered to the router and the client.
The issue only occurred in Go applications because "port_send" is overloaded
only in Go. To reproduce it, send multiple concurrent requests to the
application after it has initialised. The warning message "[unit] [go] port
NNN:dd not found" is the first visible aspect of the issue; the second and more
valuable one is a closed connection, an error response, or a hanging response to
some requests.
When the application starts, it is unaware of the router's worker thread ports,
so it requests the ports from the router after receiving requests from the
corresponding router worker threads. When multiple requests are processed
simultaneously, the router port may be required by several requests, so request
processing starts only after the application receives the required port
information. The port should be added to the Go port repository after its
'ready' flag is updated. Otherwise, Unit may start processing some requests and
use the port before it is in the repository.
The issue was introduced in changeset 78836321a126.
Application shared queue only capable to pass one shared memory buffer.
The rest buffers in chain needs to be send directly to application in response
to REQ_HEADERS_AC message.
The issue can be reproduced for configurations where 'body_buffer_size' is
greater than memory segment size (10 Mb). Requests with body size greater
than 10 Mb are just `stuck` e.g. not passed to application awaiting for more
data from router.
The bug was introduced in 1d84b9e4b459 (v1.19.0).
ASGI threads read all the requests from the queue before start processing it.
This why test need to wait a little to let the ASGI thread start request
processing and block. In virtual environment any thread or process may
be delayed and only method to avoid racing is a reasonable sleep increase.
The PORT_ACK message is the router's response to the application's NEW_PORT
message. After receiving PORT_ACK, the application is safe to process requests
using this port.
This message avoids a racing condition when the application starts processing a
request from the shared queue and sends REQ_HEADERS_ACK. The REQ_HEADERS_ACK
message contains the application port ID as reply_port, which the router uses
to send request data. When the application creates a new port, it
immediately sends it to the main router thread. Because the request is
processed outside the main thread, a racing condition can occur between the
receipt of the new port in the main thread and the receipt of REQ_HEADERS_ACK
in the worker router thread where the same port is specified as reply_port.
This fixes the router's crash on buildbot; the reason was an unexpected 'last'
response from the application to the router arriving before the response
headers. The last buffer is not a memory buffer, so the result of accessing
memory fields is unpredictable.
The unexpected 'last' message was caused by an error in libunit; fixed in
fee8fd855a00.
Data in the queue and the socket are transmitted independently; special
READ_QUEUE and READ_SOCKET message types are used for synchronization.
The warning was accidentally committed with changeset 1d84b9e4b459.
The connection's local socket address is allocated from the connection
pool before the request is passed to the application; however, with keep-alive
connections, this field was unconditionally reset by a socket configuration
value that could be NULL. For the next request, the address was allocated
again from the same connection pool. Nonetheless, all leaked addresses
were released when the connection was closed.
The issue introduced in changeset 5c7dd85fabd5.
Large-bodied requests are added to the request hash to be found when the body
arrives. However, changeset 1d84b9e4b459 introduced a bug: the 'in_hash' flag,
used to remove the request from the hash at request release, was cleared after
the first successful request lookup. As a result, the entry was never removed.
This change aids heap usage analysis in applications.
The alloc and free functions are also required for lvlhash due to the upcoming
threading support, because using main nxt_memalign() and nxt_free() isn't safe
in a multithreaded app environment. The reason is that these functions may use
thread-local structures which aren't initialized properly in applications.
The issue was introduced in changeset 1d84b9e4b459. The request buffer was
transferred via the shared application queue, but the buffer position and the
'sent' flag were not updated after the buffer had been sent.
The Apache HTTP server benchmarking tool, ab, issues HTTP/1.0 requests with
the 'Connection: Keep-Alive' header and expects a 'Connection: Keep-Alive'
header in the response.
This is required for more flexible Python version check since ASGI works
for Python 3.5+. Version check via 'startswith()' function removed as
not consistent.
Now it is possible to specify the name of the application callable using
optional parameter 'callable'. Default value is 'application'.
This closes#290 issue on GitHub.
Buffer for application prefork request allocated from temp conf mem_pool.
If error response from main process received before buffer completion handler,
temp conf mem_pool destroyed and router may crash in completion handler.
Assertion "src/nxt_buf.c:208 assertion failed: data == b->parent" triggered
when NXT_DEBUG_ALLOC enabled in configure.
This patch disables completion handler and memory allocated for buffer
released with memory pool.
The default libunit behavior relies on blocking the recv() call for port file
descriptors, which an application may override if needed. For external
applications, port file descriptors were toggled to blocking mode before the
exec() call. If the exec() call failed, descriptor remained blocked, so the
process hanged while trying to read from it.
This patch moves file descriptor mode switch inside libunit.
This is a temporary solution after the 'shared port' patch. The application
process becomes idle immediately after creation. Even if it starts processing
a request (without acknowledging it yet), it is stopped by the router because
an 'out-of-idle-time' event occurs.
Generic process-to-process shared memory exchange is no more required. Here,
it is transformed into a router-to-application pattern. The outgoing shared
memory segments collection is now the property of the application structure.
The applications connect to the router only, and the process only needs to group
the ports.
This is the port shared between all application processes which use it to pass
requests for processing. Using it significantly simplifies the request
processing code in the router. The drawback is 2 more file descriptors per each
configured application and more complex libunit message wait/read code.
The application process needs to request the shared memory segment from the
router instead of the latter pushing the segment before sending a request to
the application. This is required to simplify the communication between the
router and the application and to prepare the router for using the application
shared port and then the queue.
The application process needs to request the port from the router instead of the
latter pushing the port before sending a request to the application. This is
required to simplify the communication between the router and the application
and to prepare the router to use the application shared port and then the queue.
The goal is to minimize the number of (pid, id) to port hash lookups which
require a library mutex lock. The response port is found once per request,
while the read port is initialized at startup.
- Changed the port management callbacks to notifications, which e. g. avoids
the need to call the libunit function
- Added context and library instance reference counts for a safer resource
release
- Added the router main port initialization
If there are no listen sockets, the router configuration usage counter
remains 0 and never decreases. The only moment to release a configuration is
right after a configuration update.
According to libuv documentation, uv_poll_t memory should be released
in a callback function passed to uv_close(). Otherwise, the Node.js application
process may crash at exit.
A connection object is allocated in advance for each listen event object to be
used for the established connection. This connection needs to be freed when the
listen event is destroyed.
There is no restrictions on configration size and using segmented shared memory
only doubles memory usage because to parse configration on router side,
it needs to be 'plain' e. g. located in single continous memory buffer.
Currently, the router exits without waiting for the worker threads to stop.
There is a short gap between the runtime memory pool's free and the exit, during
which a worker thread may try to access a runtime structure. In turn, this may
cause a crash. For now, it is better to keep this memory allocated.
Max Romanov