original development tree for Linux kernel GTP module; now long in mainline.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

363 lines
9.4 KiB

fs: scale files_lock fs: scale files_lock Improve scalability of files_lock by adding per-cpu, per-sb files lists, protected with an lglock. The lglock provides fast access to the per-cpu lists to add and remove files. It also provides a snapshot of all the per-cpu lists (although this is very slow). One difficulty with this approach is that a file can be removed from the list by another CPU. We must track which per-cpu list the file is on with a new variale in the file struct (packed into a hole on 64-bit archs). Scalability could suffer if files are frequently removed from different cpu's list. However loads with frequent removal of files imply short interval between adding and removing the files, and the scheduler attempts to avoid moving processes too far away. Also, even in the case of cross-CPU removal, the hardware has much more opportunity to parallelise cacheline transfers with N cachelines than with 1. A worst-case test of 1 CPU allocating files subsequently being freed by N CPUs degenerates to contending on a single lock, which is no worse than before. When more than one CPU are allocating files, even if they are always freed by different CPUs, there will be more parallelism than the single-lock case. Testing results: On a 2 socket, 8 core opteron, I measure the number of times the lock is taken to remove the file, the number of times it is removed by the same CPU that added it, and the number of times it is removed by the same node that added it. Booting: locks= 25049 cpu-hits= 23174 (92.5%) node-hits= 23945 (95.6%) kbuild -j16 locks=2281913 cpu-hits=2208126 (96.8%) node-hits=2252674 (98.7%) dbench 64 locks=4306582 cpu-hits=4287247 (99.6%) node-hits=4299527 (99.8%) So a file is removed from the same CPU it was added by over 90% of the time. It remains within the same node 95% of the time. Tim Chen ran some numbers for a 64 thread Nehalem system performing a compile. throughput 2.6.34-rc2 24.5 +patch 24.9 us sys idle IO wait (in %) 2.6.34-rc2 51.25 28.25 17.25 3.25 +patch 53.75 18.5 19 8.75 So significantly less CPU time spent in kernel code, higher idle time and slightly higher throughput. Single threaded performance difference was within the noise of microbenchmarks. That is not to say penalty does not exist, the code is larger and more memory accesses required so it will be slightly slower. Cc: linux-kernel@vger.kernel.org Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Signed-off-by: Nick Piggin <npiggin@kernel.dk> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
11 years ago
fs: scale files_lock fs: scale files_lock Improve scalability of files_lock by adding per-cpu, per-sb files lists, protected with an lglock. The lglock provides fast access to the per-cpu lists to add and remove files. It also provides a snapshot of all the per-cpu lists (although this is very slow). One difficulty with this approach is that a file can be removed from the list by another CPU. We must track which per-cpu list the file is on with a new variale in the file struct (packed into a hole on 64-bit archs). Scalability could suffer if files are frequently removed from different cpu's list. However loads with frequent removal of files imply short interval between adding and removing the files, and the scheduler attempts to avoid moving processes too far away. Also, even in the case of cross-CPU removal, the hardware has much more opportunity to parallelise cacheline transfers with N cachelines than with 1. A worst-case test of 1 CPU allocating files subsequently being freed by N CPUs degenerates to contending on a single lock, which is no worse than before. When more than one CPU are allocating files, even if they are always freed by different CPUs, there will be more parallelism than the single-lock case. Testing results: On a 2 socket, 8 core opteron, I measure the number of times the lock is taken to remove the file, the number of times it is removed by the same CPU that added it, and the number of times it is removed by the same node that added it. Booting: locks= 25049 cpu-hits= 23174 (92.5%) node-hits= 23945 (95.6%) kbuild -j16 locks=2281913 cpu-hits=2208126 (96.8%) node-hits=2252674 (98.7%) dbench 64 locks=4306582 cpu-hits=4287247 (99.6%) node-hits=4299527 (99.8%) So a file is removed from the same CPU it was added by over 90% of the time. It remains within the same node 95% of the time. Tim Chen ran some numbers for a 64 thread Nehalem system performing a compile. throughput 2.6.34-rc2 24.5 +patch 24.9 us sys idle IO wait (in %) 2.6.34-rc2 51.25 28.25 17.25 3.25 +patch 53.75 18.5 19 8.75 So significantly less CPU time spent in kernel code, higher idle time and slightly higher throughput. Single threaded performance difference was within the noise of microbenchmarks. That is not to say penalty does not exist, the code is larger and more memory accesses required so it will be slightly slower. Cc: linux-kernel@vger.kernel.org Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Andi Kleen <ak@linux.intel.com> Signed-off-by: Nick Piggin <npiggin@kernel.dk> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
r/o bind mounts: filesystem helpers for custom 'struct file's Why do we need r/o bind mounts? This feature allows a read-only view into a read-write filesystem. In the process of doing that, it also provides infrastructure for keeping track of the number of writers to any given mount. This has a number of uses. It allows chroots to have parts of filesystems writable. It will be useful for containers in the future because users may have root inside a container, but should not be allowed to write to somefilesystems. This also replaces patches that vserver has had out of the tree for several years. It allows security enhancement by making sure that parts of your filesystem read-only (such as when you don't trust your FTP server), when you don't want to have entire new filesystems mounted, or when you want atime selectively updated. I've been using the following script to test that the feature is working as desired. It takes a directory and makes a regular bind and a r/o bind mount of it. It then performs some normal filesystem operations on the three directories, including ones that are expected to fail, like creating a file on the r/o mount. This patch: Some filesystems forego the vfs and may_open() and create their own 'struct file's. This patch creates a couple of helper functions which can be used by these filesystems, and will provide a unified place which the r/o bind mount code may patch. Also, rename an existing, static-scope init_file() to a less generic name. Signed-off-by: Dave Hansen <haveblue@us.ibm.com> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
14 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
fs: allow for more than 2^31 files Robin Holt tried to boot a 16TB system and found af_unix was overflowing a 32bit value : <quote> We were seeing a failure which prevented boot. The kernel was incapable of creating either a named pipe or unix domain socket. This comes down to a common kernel function called unix_create1() which does: atomic_inc(&unix_nr_socks); if (atomic_read(&unix_nr_socks) > 2 * get_max_files()) goto out; The function get_max_files() is a simple return of files_stat.max_files. files_stat.max_files is a signed integer and is computed in fs/file_table.c's files_init(). n = (mempages * (PAGE_SIZE / 1024)) / 10; files_stat.max_files = n; In our case, mempages (total_ram_pages) is approx 3,758,096,384 (0xe0000000). That leaves max_files at approximately 1,503,238,553. This causes 2 * get_max_files() to integer overflow. </quote> Fix is to let /proc/sys/fs/file-nr & /proc/sys/fs/file-max use long integers, and change af_unix to use an atomic_long_t instead of atomic_t. get_max_files() is changed to return an unsigned long. get_nr_files() is changed to return a long. unix_nr_socks is changed from atomic_t to atomic_long_t, while not strictly needed to address Robin problem. Before patch (on a 64bit kernel) : # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max -18446744071562067968 After patch: # echo 2147483648 >/proc/sys/fs/file-max # cat /proc/sys/fs/file-max 2147483648 # cat /proc/sys/fs/file-nr 704 0 2147483648 Reported-by: Robin Holt <holt@sgi.com> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com> Acked-by: David Miller <davem@davemloft.net> Reviewed-by: Robin Holt <holt@sgi.com> Tested-by: Robin Holt <holt@sgi.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
11 years ago
  1. /*
  2. * linux/fs/file_table.c
  3. *
  4. * Copyright (C) 1991, 1992 Linus Torvalds
  5. * Copyright (C) 1997 David S. Miller (davem@caip.rutgers.edu)
  6. */
  7. #include <linux/string.h>
  8. #include <linux/slab.h>
  9. #include <linux/file.h>
  10. #include <linux/fdtable.h>
  11. #include <linux/init.h>
  12. #include <linux/module.h>
  13. #include <linux/fs.h>
  14. #include <linux/security.h>
  15. #include <linux/eventpoll.h>
  16. #include <linux/rcupdate.h>
  17. #include <linux/mount.h>
  18. #include <linux/capability.h>
  19. #include <linux/cdev.h>
  20. #include <linux/fsnotify.h>
  21. #include <linux/sysctl.h>
  22. #include <linux/lglock.h>
  23. #include <linux/percpu_counter.h>
  24. #include <linux/percpu.h>
  25. #include <linux/hardirq.h>
  26. #include <linux/task_work.h>
  27. #include <linux/ima.h>
  28. #include <linux/atomic.h>
  29. #include "internal.h"
  30. /* sysctl tunables... */
  31. struct files_stat_struct files_stat = {
  32. .max_files = NR_FILE
  33. };
  34. /* SLAB cache for file structures */
  35. static struct kmem_cache *filp_cachep __read_mostly;
  36. static struct percpu_counter nr_files __cacheline_aligned_in_smp;
  37. static void file_free_rcu(struct rcu_head *head)
  38. {
  39. struct file *f = container_of(head, struct file, f_u.fu_rcuhead);
  40. put_cred(f->f_cred);
  41. kmem_cache_free(filp_cachep, f);
  42. }
  43. static inline void file_free(struct file *f)
  44. {
  45. percpu_counter_dec(&nr_files);
  46. file_check_state(f);
  47. call_rcu(&f->f_u.fu_rcuhead, file_free_rcu);
  48. }
  49. /*
  50. * Return the total number of open files in the system
  51. */
  52. static long get_nr_files(void)
  53. {
  54. return percpu_counter_read_positive(&nr_files);
  55. }
  56. /*
  57. * Return the maximum number of open files in the system
  58. */
  59. unsigned long get_max_files(void)
  60. {
  61. return files_stat.max_files;
  62. }
  63. EXPORT_SYMBOL_GPL(get_max_files);
  64. /*
  65. * Handle nr_files sysctl
  66. */
  67. #if defined(CONFIG_SYSCTL) && defined(CONFIG_PROC_FS)
  68. int proc_nr_files(ctl_table *table, int write,
  69. void __user *buffer, size_t *lenp, loff_t *ppos)
  70. {
  71. files_stat.nr_files = get_nr_files();
  72. return proc_doulongvec_minmax(table, write, buffer, lenp, ppos);
  73. }
  74. #else
  75. int proc_nr_files(ctl_table *table, int write,
  76. void __user *buffer, size_t *lenp, loff_t *ppos)
  77. {
  78. return -ENOSYS;
  79. }
  80. #endif
  81. /* Find an unused file structure and return a pointer to it.
  82. * Returns an error pointer if some error happend e.g. we over file
  83. * structures limit, run out of memory or operation is not permitted.
  84. *
  85. * Be very careful using this. You are responsible for
  86. * getting write access to any mount that you might assign
  87. * to this filp, if it is opened for write. If this is not
  88. * done, you will imbalance int the mount's writer count
  89. * and a warning at __fput() time.
  90. */
  91. struct file *get_empty_filp(void)
  92. {
  93. const struct cred *cred = current_cred();
  94. static long old_max;
  95. struct file *f;
  96. int error;
  97. /*
  98. * Privileged users can go above max_files
  99. */
  100. if (get_nr_files() >= files_stat.max_files && !capable(CAP_SYS_ADMIN)) {
  101. /*
  102. * percpu_counters are inaccurate. Do an expensive check before
  103. * we go and fail.
  104. */
  105. if (percpu_counter_sum_positive(&nr_files) >= files_stat.max_files)
  106. goto over;
  107. }
  108. f = kmem_cache_zalloc(filp_cachep, GFP_KERNEL);
  109. if (unlikely(!f))
  110. return ERR_PTR(-ENOMEM);
  111. percpu_counter_inc(&nr_files);
  112. f->f_cred = get_cred(cred);
  113. error = security_file_alloc(f);
  114. if (unlikely(error)) {
  115. file_free(f);
  116. return ERR_PTR(error);
  117. }
  118. atomic_long_set(&f->f_count, 1);
  119. rwlock_init(&f->f_owner.lock);
  120. spin_lock_init(&f->f_lock);
  121. eventpoll_init_file(f);
  122. /* f->f_version: 0 */
  123. return f;
  124. over:
  125. /* Ran out of filps - report that */
  126. if (get_nr_files() > old_max) {
  127. pr_info("VFS: file-max limit %lu reached\n", get_max_files());
  128. old_max = get_nr_files();
  129. }
  130. return ERR_PTR(-ENFILE);
  131. }
  132. /**
  133. * alloc_file - allocate and initialize a 'struct file'
  134. * @mnt: the vfsmount on which the file will reside
  135. * @dentry: the dentry representing the new file
  136. * @mode: the mode with which the new file will be opened
  137. * @fop: the 'struct file_operations' for the new file
  138. *
  139. * Use this instead of get_empty_filp() to get a new
  140. * 'struct file'. Do so because of the same initialization
  141. * pitfalls reasons listed for init_file(). This is a
  142. * preferred interface to using init_file().
  143. *
  144. * If all the callers of init_file() are eliminated, its
  145. * code should be moved into this function.
  146. */
  147. struct file *alloc_file(struct path *path, fmode_t mode,
  148. const struct file_operations *fop)
  149. {
  150. struct file *file;
  151. file = get_empty_filp();
  152. if (IS_ERR(file))
  153. return file;
  154. file->f_path = *path;
  155. file->f_inode = path->dentry->d_inode;
  156. file->f_mapping = path->dentry->d_inode->i_mapping;
  157. file->f_mode = mode;
  158. file->f_op = fop;
  159. /*
  160. * These mounts don't really matter in practice
  161. * for r/o bind mounts. They aren't userspace-
  162. * visible. We do this for consistency, and so
  163. * that we can do debugging checks at __fput()
  164. */
  165. if ((mode & FMODE_WRITE) && !special_file(path->dentry->d_inode->i_mode)) {
  166. file_take_write(file);
  167. WARN_ON(mnt_clone_write(path->mnt));
  168. }
  169. if ((mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
  170. i_readcount_inc(path->dentry->d_inode);
  171. return file;
  172. }
  173. EXPORT_SYMBOL(alloc_file);
  174. /**
  175. * drop_file_write_access - give up ability to write to a file
  176. * @file: the file to which we will stop writing
  177. *
  178. * This is a central place which will give up the ability
  179. * to write to @file, along with access to write through
  180. * its vfsmount.
  181. */
  182. static void drop_file_write_access(struct file *file)
  183. {
  184. struct vfsmount *mnt = file->f_path.mnt;
  185. struct dentry *dentry = file->f_path.dentry;
  186. struct inode *inode = dentry->d_inode;
  187. put_write_access(inode);
  188. if (special_file(inode->i_mode))
  189. return;
  190. if (file_check_writeable(file) != 0)
  191. return;
  192. __mnt_drop_write(mnt);
  193. file_release_write(file);
  194. }
  195. /* the real guts of fput() - releasing the last reference to file
  196. */
  197. static void __fput(struct file *file)
  198. {
  199. struct dentry *dentry = file->f_path.dentry;
  200. struct vfsmount *mnt = file->f_path.mnt;
  201. struct inode *inode = file->f_inode;
  202. might_sleep();
  203. fsnotify_close(file);
  204. /*
  205. * The function eventpoll_release() should be the first called
  206. * in the file cleanup chain.
  207. */
  208. eventpoll_release(file);
  209. locks_remove_flock(file);
  210. if (unlikely(file->f_flags & FASYNC)) {
  211. if (file->f_op->fasync)
  212. file->f_op->fasync(-1, file, 0);
  213. }
  214. ima_file_free(file);
  215. if (file->f_op->release)
  216. file->f_op->release(inode, file);
  217. security_file_free(file);
  218. if (unlikely(S_ISCHR(inode->i_mode) && inode->i_cdev != NULL &&
  219. !(file->f_mode & FMODE_PATH))) {
  220. cdev_put(inode->i_cdev);
  221. }
  222. fops_put(file->f_op);
  223. put_pid(file->f_owner.pid);
  224. if ((file->f_mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ)
  225. i_readcount_dec(inode);
  226. if (file->f_mode & FMODE_WRITE)
  227. drop_file_write_access(file);
  228. file->f_path.dentry = NULL;
  229. file->f_path.mnt = NULL;
  230. file->f_inode = NULL;
  231. file_free(file);
  232. dput(dentry);
  233. mntput(mnt);
  234. }
  235. static LLIST_HEAD(delayed_fput_list);
  236. static void delayed_fput(struct work_struct *unused)
  237. {
  238. struct llist_node *node = llist_del_all(&delayed_fput_list);
  239. struct llist_node *next;
  240. for (; node; node = next) {
  241. next = llist_next(node);
  242. __fput(llist_entry(node, struct file, f_u.fu_llist));
  243. }
  244. }
  245. static void ____fput(struct callback_head *work)
  246. {
  247. __fput(container_of(work, struct file, f_u.fu_rcuhead));
  248. }
  249. /*
  250. * If kernel thread really needs to have the final fput() it has done
  251. * to complete, call this. The only user right now is the boot - we
  252. * *do* need to make sure our writes to binaries on initramfs has
  253. * not left us with opened struct file waiting for __fput() - execve()
  254. * won't work without that. Please, don't add more callers without
  255. * very good reasons; in particular, never call that with locks
  256. * held and never call that from a thread that might need to do
  257. * some work on any kind of umount.
  258. */
  259. void flush_delayed_fput(void)
  260. {
  261. delayed_fput(NULL);
  262. }
  263. static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput);
  264. void fput(struct file *file)
  265. {
  266. if (atomic_long_dec_and_test(&file->f_count)) {
  267. struct task_struct *task = current;
  268. if (likely(!in_interrupt() && !(task->flags & PF_KTHREAD))) {
  269. init_task_work(&file->f_u.fu_rcuhead, ____fput);
  270. if (!task_work_add(task, &file->f_u.fu_rcuhead, true))
  271. return;
  272. /*
  273. * After this task has run exit_task_work(),
  274. * task_work_add() will fail. Fall through to delayed
  275. * fput to avoid leaking *file.
  276. */
  277. }
  278. if (llist_add(&file->f_u.fu_llist, &delayed_fput_list))
  279. schedule_delayed_work(&delayed_fput_work, 1);
  280. }
  281. }
  282. /*
  283. * synchronous analog of fput(); for kernel threads that might be needed
  284. * in some umount() (and thus can't use flush_delayed_fput() without
  285. * risking deadlocks), need to wait for completion of __fput() and know
  286. * for this specific struct file it won't involve anything that would
  287. * need them. Use only if you really need it - at the very least,
  288. * don't blindly convert fput() by kernel thread to that.
  289. */
  290. void __fput_sync(struct file *file)
  291. {
  292. if (atomic_long_dec_and_test(&file->f_count)) {
  293. struct task_struct *task = current;
  294. BUG_ON(!(task->flags & PF_KTHREAD));
  295. __fput(file);
  296. }
  297. }
  298. EXPORT_SYMBOL(fput);
  299. void put_filp(struct file *file)
  300. {
  301. if (atomic_long_dec_and_test(&file->f_count)) {
  302. security_file_free(file);
  303. file_free(file);
  304. }
  305. }
  306. void __init files_init(unsigned long mempages)
  307. {
  308. unsigned long n;
  309. filp_cachep = kmem_cache_create("filp", sizeof(struct file), 0,
  310. SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
  311. /*
  312. * One file with associated inode and dcache is very roughly 1K.
  313. * Per default don't use more than 10% of our memory for files.
  314. */
  315. n = (mempages * (PAGE_SIZE / 1024)) / 10;
  316. files_stat.max_files = max_t(unsigned long, n, NR_FILE);
  317. files_defer_init();
  318. percpu_counter_init(&nr_files, 0);
  319. }