@JanMikes and @tagur87 on GitHub both reported issues with long URLs
that were exceeding the 8192 byte large_header_buffer_size setting,
which resulted in a HTTP 431 error (Request Header Fields Too Large).
This can be resolved in the code by updating the following line in
src/nxt_router.c::nxt_router_conf_create()
skcf->large_header_buffer_size = 8192;
However, requiring users to modify unit and install custom versions is
less than ideal. We could increase the value, but to what?
This commit takes the option of allowing the user to set this option in
their config by making large_header_buffer_size a valid configuration
setting.
large_header_buffer_size is already set by the configuration system in
nxt_router.c it just isn't set as a valid config option in
nxt_conf_validation.c
With this change users can set this option in their config if required
by the following
"settings": {
"http": {
"large_header_buffer_size": 16384
}
},
It retains its default value of 8192 bytes if this is not set.
With this commit, without the above setting or too low a value, with a
long URL you get a 431 error. With the above setting set to a large
enough value, the request is successful.
NOTE: This setting really determines the maximum size of any single
header _value_. Also, unit will try and place multiple values
into a buffer _if_ they fully fit.
NOTE: This is being released as undocumented and subject to change as it
exposes internal workings of unit.
Closes: <https://github.com/nginx/unit/issues/521>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
This hooks the cgroup support up to the config system so it can actually
be used.
To make use of this in unit a new "cgroup" section has been added to the
isolation configuration.
e.g
"applications": {
"python": {
"type": "python",
"processes": 5,
"path": "/opt/unit/unit-cgroup-test/",
"module": "app",
"isolation": {
"cgroup": {
"path": "app/python"
}
}
}
}
Now there are two ways to specify the path, relative, like the above
(without a leading '/') and absolute (with a leading '/').
In the above case the "python" application is placed into its own cgroup
under CGROUP_ROOT/<main unit process cgroup>/app/python. Whereas if you
specified say
"path": "/unit/app/python"
Then the python application would be placed under
CGROUP_ROOT/unit/app/python
The first option allows you to easily take advantage of any resource
limits that have already been configured for unit.
With the second method (absolute pathname) if you know of an already
existing cgroup where you'd like to place it, you can, e.g
"path": "/system.slice/unit/python"
Where system.slice has already been created by systemd and may already
have some overall system limits applied which would also apply to unit.
Limits apply down the hierarchy and lower groups can't exceed the
previous group limits.
So what does this actually look like? Lets take the unit-calculator
application[0] and have each of its applications placed into their own
cgroup. If we give each application a new section like
"isolation": {
"cgroup": {
"path": "/unit/unit-calculator/add"
}
}
changing the path for each one, we can visualise the result with the
systemd-cgls command, e.g
│ └─session-5.scope (#4561)
│ ├─ 6667 sshd: andrew [priv]
│ ├─ 6684 sshd: andrew@pts/0
│ ├─ 6685 -bash
│ ├─ 12632 unit: main v1.28.0 [/opt/unit/sbin/unitd --control 127.0.0.1:808>
│ ├─ 12634 unit: controller
│ ├─ 12635 unit: router
│ ├─ 13550 systemd-cgls
│ └─ 13551 less
├─unit (#4759)
│ └─unit-calculator (#5037)
│ ├─subtract (#5069)
│ │ ├─ 12650 unit: "subtract" prototype
│ │ └─ 12651 unit: "subtract" application
│ ├─multiply (#5085)
│ │ ├─ 12653 unit: "multiply" prototype
│ │ └─ 12654 unit: "multiply" application
│ ├─divide (#5101)
│ │ ├─ 12671 unit: "divide" prototype
│ │ └─ 12672 node divide.js
│ ├─sqroot (#5117)
│ │ ├─ 12679 unit: "sqroot" prototype
│ │ └─ 12680 /home/andrew/src/unit-calculator/sqroot/sqroot
│ └─add (#5053)
│ ├─ 12648 unit: "add" prototype
│ └─ 12649 unit: "add" application
We used an absolute path so the cgroups will be created relative to the
main cgroupfs mount, e.g /sys/fs/cgroup
We can see that the main unit processes are in the same cgroup as the
shell from where they were started, by default child process are placed
into the same cgroup as the parent.
Then we can see that each application has been placed into its own
cgroup under /sys/fs/cgroup
Taking another example of a simple 5 process python application, with
"isolation": {
"cgroup": {
"path": "app/python"
}
}
Here we have specified a relative path and thus the python application
will be placed below the existing cgroup that contains the main unit
process. E.g
│ │ │ ├─app-glib-cinnamon\x2dcustom\x2dlauncher\x2d3-43951.scope (#90951)
│ │ │ │ ├─ 988 unit: main v1.28.0 [/opt/unit/sbin/unitd --no-daemon]
│ │ │ │ ├─ 990 unit: controller
│ │ │ │ ├─ 991 unit: router
│ │ │ │ ├─ 43951 xterm -bg rgb:20/20/20 -fg white -fa DejaVu Sans Mono
│ │ │ │ ├─ 43956 bash
│ │ │ │ ├─ 58828 sudo -i
│ │ │ │ ├─ 58831 -bash
│ │ │ │ └─app (#107351)
│ │ │ │ └─python (#107367)
│ │ │ │ ├─ 992 unit: "python" prototype
│ │ │ │ ├─ 993 unit: "python" application
│ │ │ │ ├─ 994 unit: "python" application
│ │ │ │ ├─ 995 unit: "python" application
│ │ │ │ ├─ 996 unit: "python" application
│ │ │ │ └─ 997 unit: "python" application
[0]: <https://github.com/lcrilly/unit-calculator>
Reviewed-by: Alejandro Colomar <alx@nginx.com>
Signed-off-by: Andrew Clayton <a.clayton@nginx.com>
It's for the introduction of njs support.
For each option that supports native variable and JS template literals introduced next,
it's unified as template string.
No functional changes.
The casts are unnecessary, since memchr(3)'s argument is 'const void *'.
It might have been necessary in the times of K&R, where 'void *' didn't
exist. Nowadays, it's unnecessary, and _very_ unsafe, since casts can
hide all classes of bugs by silencing most compiler warnings.
The changes from nxt_memchr() to memchr(3) were scripted:
$ find src/ -type f \
| grep '\.[ch]$' \
| xargs sed -i 's/nxt_memchr/memchr/'
Reviewed-by: Andrew Clayton <a.clayton@nginx.com>
Signed-off-by: Alejandro Colomar <alx@nginx.com>
This commit removed the $uri auto-append for the "share" option
introduced in rev be6409cdb028.
The main reason is that it causes problems when preparing Unit configurations
to be loaded at startup from the state directory. E.g. Docker. A valid conf.json
file with $uri references will end up with $uri$uri due to the auto-append.
We accept both "\u0000socket-name" and "@socket-name" as abstract
unix sockets. The first one is passed to the kernel pristine,
while the second is transformed '@'->'\0'.
The commit that added support for unix sockets accepts both
variants, but we internally stored it in the same way, using
"\u0000..." for both.
We want to support abstract sockets transparently to the user, so
that if the user configures unitd with '@', if we receive a query
about the current configuration, the user should see the same
exact thing that was configured. So, this commit avoids the
transformation in the internal state file, storing user input
pristine, and we only transform the '@' for a string that will
be used internally (not user-visible).
This commit (indirectly) fixes a small bug, where we created
abstract sockets with a trailing '\0' in their name due to calling
twice nxt_sockaddr_parse() on the same string. By calling that
function only once with each copy of the string, we have fixed that
bug.
This supports a new option "index" that configures a custom index
file name to be served when a directory is requested. This
initial support only allows a single fixed string. An example:
{
"share": "/www/data/static/$uri",
"index": "lookatthis.htm"
}
When <example.com/foo/bar/> is requested,
</www/data/static/foo/bar/lookatthis.html> is served.
Default is "index.html".
===
nxt_conf_validator.c:
Accept "index" as a member of "share", and make sure it's a string.
===
I tried this feature in my own computer, where I tried the
following:
- Setting "index" to "lookatthis.htm", and check that the correct
file is being served (check both a different name and a
different extension).
- Not setting "index", and check that <index.html> is being
served.
- Settind "index" to an array of strings, and check that the
configuration fails:
{
"error": "Invalid configuration.",
"detail": "The \"index\" value must be a string, but not an array."
}
............
Description:
............
Before this commit, the encoded URI could be calculated at
configuration time. Now, since variables can only be resolved at
request time, we have different situations:
- "location" contains no variables:
In this case, we still encode the URI in the conf structure, at
configuration time, and then we just copy the resulting string
to the ctx structure at request time.
- "location" contains variables:
In this case, we compile the var string at configure time, then
when we resolve it at request time, and then we encode the
string.
In both cases, as was being done before, if the string is empty,
either before or after resolving variables, we skip the encoding.
...........
Usefulness:
...........
An example of why this feature may be useful is redirecting HTTP
to HTTPS with something like:
"action": {
"return": 301,
"location": "https://${host}${uri}"
}
.....
Bugs:
.....
This feature conflicts with the relevant RFCs in the following:
'$' is used for Unit variables, but '$' is a reserved character in
a URI, to be used as a sub-delimiter. However, it's almost never
used as that, and in fact, other parts of Unit already conflict
with '$' being a reserved character for use as a sub-delimiter, so
this is at least consistent in that sense. VBart suggested an
easy workaround if we ever need it: adding a variable '$sign'
which resolves to a literal '$'.
......
Notes:
......
An empty string is handled as if "location" wasn't specified at
all, so no Location header is sent.
This is incorrect, and the code is slightly misleading.
The Location header consists of a URI-reference[1], which might be
a relative one, which itself might consist of an empty string[2].
[1]: <https://www.rfc-editor.org/rfc/rfc7231#section-7.1.2>
[2]: <https://stackoverflow.com/a/43338457>
Now that we have variables, it's more likely that an empty
Location header will be requested, and we should handle it
correctly.
I think in a future commit we should modify the code to allow
differentiating between an unset "location" and an empty one,
which should be treated as any other "location" string.
.................
Testing (manual):
.................
{
"listeners": {
"*:80": {
"pass": "routes/str"
},
"*:81": {
"pass": "routes/empty"
},
"*:82": {
"pass": "routes/var"
},
"*:83": {
"pass": "routes/enc-str"
},
"*:84": {
"pass": "routes/enc-var"
}
},
"routes": {
"str": [
{
"action": {
"return": 301,
"location": "foo"
}
}
],
"empty": [
{
"action": {
"return": 301,
"location": ""
}
}
],
"var": [
{
"action": {
"return": 301,
"location": "$host"
}
}
],
"enc-str": [
{
"action": {
"return": 301,
"location": "f%23o#o"
}
}
],
"enc-var": [
{
"action": {
"return": 301,
"location": "f%23o${host}#o"
}
}
]
}
}
$ curl --dump-header - localhost:80
HTTP/1.1 301 Moved Permanently
Location: foo
Server: Unit/1.27.0
Date: Thu, 07 Apr 2022 23:30:06 GMT
Content-Length: 0
$ curl --dump-header - localhost:81
HTTP/1.1 301 Moved Permanently
Server: Unit/1.27.0
Date: Thu, 07 Apr 2022 23:30:08 GMT
Content-Length: 0
$ curl --dump-header - localhost:82
HTTP/1.1 301 Moved Permanently
Location: localhost
Server: Unit/1.27.0
Date: Thu, 07 Apr 2022 23:30:15 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: bar" localhost:82
HTTP/1.1 301 Moved Permanently
Location: bar
Server: Unit/1.27.0
Date: Thu, 07 Apr 2022 23:30:23 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: " localhost:82
HTTP/1.1 301 Moved Permanently
Server: Unit/1.27.0
Date: Thu, 07 Apr 2022 23:30:29 GMT
Content-Length: 0
$ curl --dump-header - localhost:83
HTTP/1.1 301 Moved Permanently
Location: f%23o#o
Server: Unit/1.27.0
Date: Sat, 09 Apr 2022 11:22:23 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: " localhost:84
HTTP/1.1 301 Moved Permanently
Location: f%23o#o
Server: Unit/1.27.0
Date: Sat, 09 Apr 2022 11:22:44 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: alx" localhost:84
HTTP/1.1 301 Moved Permanently
Location: f%23oalx#o
Server: Unit/1.27.0
Date: Sat, 09 Apr 2022 11:22:52 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: a#l%23x" localhost:84
HTTP/1.1 301 Moved Permanently
Location: f%2523oa#l%2523x%23o
Server: Unit/1.27.0
Date: Sat, 09 Apr 2022 11:23:09 GMT
Content-Length: 0
$ curl --dump-header - -H "Host: b##ar" localhost:82
HTTP/1.1 301 Moved Permanently
Location: b#%23ar
Server: Unit/1.27.0
Date: Sat, 09 Apr 2022 11:25:01 GMT
Content-Length: 0
When testing some configurations of compilers and OSes, I noticed
that clang(1) 13 on Debian caused a function to be compiled but
unused, and the compiler triggered a compile error.
To avoid that error, use __attribute__((__unused__)). Let's call
our wrapper NXT_MAYBE_UNUSED, since it describes itself more
precisely than the GCC attribute name. It's also the name that
C2x (likely C23) has given to the standard attribute, which is
[[maybe_unused]], so it's also likely to be more readable because
of that name being in ISO C.
Compared to the previous implementation based on OpenSSL, the new implementation
has these advantages:
1. Strict and reliable detection of invalid strings, including strings with
less than 4 bytes of garbage at the end;
2. Allows to use Base64 strings without '=' padding.
This commit supports variable in the "share" option, the finding path to
file serve is the value from "share". An example:
{
"share": "/www/data/static$uri"
}
This commit introduces the replacement of the client address based on the value
of a specified HTTP header. This is intended for use when Unit is placed
behind a reverse proxy like nginx or a CDN.
You must specify the source addresses of the trusted proxies. This can be
accomplished with any valid IP pattern supported by Unit's match block:
["10.0.0.1", "10.4.0.0/16", "!192.168.1.1"]
The feature is configured per listener.
The client address replacement functionality only operates when there is a
source IP match and the specified header is present. Typically this would be
an 'X-Forwarded-For' header.
{
"listeners": {
"127.0.0.1:8080": {
"client_ip": {
"header": "X-Forwarded-For",
"source": [
"10.0.0.0/8"
]
},
"pass": "applications/my_app"
},
}
}
If a request occurs and Unit receives a header like below:
"X-Forwarded-For: 84.123.23.23"
By default, Unit trusts the last rightmost IP in the header, so REMOTE_ADDR
will be set to 84.123.23.23 if the connection originated from 10.0.0.0/8.
If Unit runs behind consecutive reverse proxies and receives a header similar
to the following:
"X-Forwarded-For: 84.123.23.23, 10.0.0.254"
You will need to enable "recursive" checking, which walks the header from
last address to first and chooses the first non-trusted address it finds.
{
"listeners": {
"127.0.0.1:8080": {
"client_ip": {
"header": "X-Forwarded-For",
"source": [
"10.0.0.0/8"
]
"recursive": true,
},
"pass": "applications/my_app"
},
}
}
If a connection from 10.0.0.0/8 occurs, the chain is walked. Here, 10.0.0.254
is also a trusted address so the client address will be replaced with
84.123.23.23.
If all IP addresses in the header are trusted, the client address is set to
the first address in the header:
If 10.0.0.0/8 is trusted and "X-Forwarded-For: 10.0.0.3, 10.0.0.2, 10.0.0.1",
the client address will be replaced with 10.0.0.3.
To support TLS sessions, Unit uses the OpenSSL built-in session cache; the
cache_size option defines the number sessions to store. To disable the feather,
the option must be zero.
This feature allows one to specify blocks of code that are called when certain
lifecycle events occur. A user configures a "hooks" property on the app
configuration that points to a script. This script will be evaluated on boot
and should contain blocks of code that will be called on specific events.
An example of configuration:
{
"type": "ruby",
"processes": 2,
"threads": 2,
"user": "vagrant",
"group": "vagrant",
"script": "config.ru",
"hooks": "hooks.rb",
"working_directory": "/home/vagrant/unit/rbhooks",
"environment": {
"GEM_HOME": "/home/vagrant/.ruby"
}
}
An example of a valid "hooks.rb" file follows:
File.write("./hooks.#{Process.pid}", "hooks evaluated")
on_worker_boot do
File.write("./worker_boot.#{Process.pid}", "worker booted")
end
on_thread_boot do
File.write("./thread_boot.#{Process.pid}.#{Thread.current.object_id}",
"thread booted")
end
on_thread_shutdown do
File.write("./thread_shutdown.#{Process.pid}.#{Thread.current.object_id}",
"thread shutdown")
end
on_worker_shutdown do
File.write("./worker_shutdown.#{Process.pid}", "worker shutdown")
end
This closes issue #535 on GitHub.
To perform various configuration operations on SSL_CTX, OpenSSL provides
SSL_CONF_cmd(). Specifically, to configure ciphers for a listener,
"CipherString" and "Ciphersuites" file commands are used:
https://www.openssl.org/docs/man1.1.1/man3/SSL_CONF_cmd.html
This feature can be configured in the "tls/conf_commands" section.
The certificate is selected by matching the arriving SNI to the common name and
the alternatives names. If no certificate matches the name, the first bundle in
the array is chosen.
Instead of PTHREAD_STACK_MIN define, NetBSD requires to get minimum stack
size using sysctl(_SC_THREAD_STACK_MIN).
This change originally proposed by Juraj Lutter <juraj@lutter.sk>.
Previously, all requests that contained in header field names characters other
than alphanumeric, or "-", or "_" were rejected with a 400 "Bad Request" error
response.
Now, the parser allows the same set of characters as specified in RFC 7230,
including: "!", "#", "$", "%", "&", "'", "*", "+", ".", "^", "`", "|", and "~".
Header field names that contain only these characters are considered valid.
Also, there's a new option introduced: "discard_unsafe_fields". It accepts
boolean value and it is set to "true" by default.
When this option is "true", all header field names that contain characters
in valid range, but other than alphanumeric or "-" are skipped during parsing.
When the option is "false", these header fields aren't skipped.
Requests with non-valid characters in header field names according to
RFC 7230 are rejected regardless of "discard_unsafe_fields" setting.
This closes#422 issue on GitHub.
Now it is possible to specify the name of the application callable using
optional parameter 'callable'. Default value is 'application'.
This closes#290 issue on GitHub.
