Port message handler may perform fork() and then close port read file
descriptor and enable write on same event fd. Next read attempt in this case
may cause different errors in log file.
Remove pid proxying to worker engines implementation was originally
overcomplicated. Memory pool and 2 engine posts (there and back again) are
optimized out and replaced with band new nxt_port_post() call.
Request <-> application link structure (nxt_req_app_link_t) used to register
the request in application request queue (nxt_app_t.requests) and generate
application-specific port message.
Now it is allocated from request pool. This pool created for request parsing
and used to allocate and store information specific to this request.
Request can be processed in thread different from the thread where the
connection originally handled.
Because of possible racing conditions, using original connection structures
is unsafe. To solve this, error condition is registered in 'ra' (request <->
application link) and traversed back to original connection thread where
the error message can be generated and send back to client.
For empty write queue cases, it is possible to avoid allocation and enqueue
send message structures. Send message initialized on stack and passed to
write handler. If immediate write fails, send message allocated from engine
pool and enqueued.
Use counter helps to simplify logic around port and application free.
Port 'post' function introduced to simplify post execution of particular
function to original port engine's thread.
Write message queue is protected by mutex which makes port write operation
thread safe.
To allow use port from different threads, the first step is to avoid using
port's memory pool for temporary allocations required to send data through
the port. Including but not limited by:
- buffers for data;
- send message structures;
- new mmap fd notifications;
It is still safe to use port memory pool for incoming buffers allocations
because recieve operation bound to single thread.
Worker threads ports need to receive 'remove pid' message to properly handle
application process exit case and finish requests processed by particular
application worker. Main process send 'remove pid' notification to service
thread port only and this message must be 'proxied' to other running engines.
Separate memory pool created for this message. For each engine structure
required to post message to engine allocate from the pool using 'retain'
allocation method. After successfull post structure will be freed using
'release' method. To completely destroy poll one more 'release' should be
called to release initial reference count.
I'm afraid this should be simplified using good old malloc() and free() calls.
Do not reuse shared memory segment with different port until this segment
successfully received and indexed on other side. However, segment can be used
to transfer data via the port it was sent at any time.
PHP SAPI tries to read body for POST request before registering
header-specific variables. For other methods, read_post_body() called by SAPI
after variables registration.
This closes#10 issue on GitHub.
Application free ports is a queue (double linked list) protected with mutex.
After successfull request parsing, each router thread (1) tries to get port
from this list. If this list is empty, (2) start worker request posted to main
router thread. Another thread may release port between (1) and (2).
This fix adds an attempt to get port from free ports list at the beginning of
start worker action in main thread.
Usage:
1. Register handlers in incoming port with nxt_port_rpc_register_handler().
2. Use return value as a stream identifier for next nxt_port_socket_write().
NXT_PORT_MSG_LAST - mark message as last;
NXT_PORT_MSG_CLOSE_FD - close fd right after send;
Type constants altered to include last flag for single buffer messages.
Last sign is critical for coming port RPC layer. Handlers unregistered on last
message. Create sync buffer is not convenient, extra parameter is better.
nxt_req_conn_link_t still used for lookup connection by request id.
New nxt_req_app_link_t (ra) allocated from conn->mem_pool using mp_retain().
ra stored in app->requests if there is no free worker to process request.
New port message type introduced NXT_PORT_MSG_REMOVE_PID. Default handler
removes process description from nxt_runtime_t with all ports, incoming and
outgoing mmaps etc.
There is a case in router where we use port in router connection thread.
Buffers are allocated within connection memory pool which can be used only in
this router thread. sendmsg() can be postponed into main router thread and
completion handler will compare current engine and post itself to correct
engine.
- request to connection mapping in engine;
- requests queue in connection;
- engine port creation;
- connected ports hash for each process;
- engine port data messages processing (app responses);
nxt_port_mmap_t stored in arrays and it is unsafe to store
pointer to array element.
Shared memory structures and macros moved to separate header
file to be used by GO package.
Max Romanov