</refentry>
</sect1>
+
+ <sect1 id="Raster_Tips">
+ <sect1info>
+ <abstract>
+ <para>This section documents various gotchas and tips related to PostGIS Raster.</para>
+ </abstract>
+ </sect1info>
+ <title>Raster Tips</title>
+
+ <sect2 id="outdb">
+ <title>Out-DB Rasters</title>
+
+ <sect3 id="gdal_readdir">
+ <title>Directory containing many files</title>
+ <para>
+ When GDAL opens a file, GDAL eagerly scans the directory of that file to build a catalog of other files. If this directory contains many files (e.g. thousands, millions), opening that file becomes extremely slow (especially if that file happens to be on a network drive such as NFS).
+ </para>
+ <para>
+ To control this behavior, GDAL provides the following environment variable: <ulink url="https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_DISABLE_READDIR_ON_OPEN">GDAL_DISABLE_READDIR_ON_OPEN</ulink>. Set <varname>GDAL_DISABLE_READDIR_ON_OPEN</varname> to <varname>TRUE</varname> to disable directory scanning.
+ </para>
+
+ <para>
+ In Ubuntu (and assuming you are using PostgreSQL's packages for Ubuntu), <varname>GDAL_DISABLE_READDIR_ON_OPEN</varname> can be set in <emphasis>/etc/postgresql/POSTGRESQL_VERSION/CLUSTER_NAME/environment</emphasis> (where POSTGRESQL_VERSION is the version of PostgreSQL, e.g. 9.6 and CLUSTER_NAME is the name of the cluster, e.g. maindb). You can also set PostGIS environment variables here as well.
+
+ <programlisting>
+# environment variables for postmaster process
+# This file has the same syntax as postgresql.conf:
+# VARIABLE = simple_value
+# VARIABLE2 = 'any value!'
+# I. e. you need to enclose any value which does not only consist of letters,
+# numbers, and '-', '_', '.' in single quotes. Shell commands are not
+# evaluated.
+POSTGIS_GDAL_ENABLED_DRIVERS = 'ENABLE_ALL'
+
+POSTGIS_ENABLE_OUTDB_RASTERS = 1
+
+<emphasis role="strong">GDAL_DISABLE_READDIR_ON_OPEN = 'TRUE'</emphasis>
+ </programlisting>
+ </para>
+
+ </sect3>
+
+ <sect3 id="max_open_files">
+ <title>Maximum Number of Open Files</title>
+
+ <para>
+ The maximum number of open files permitted by Linux and PostgreSQL are typically conservative (typically 1024 open files per process) given the assumption that the system is consumed by human users. For Out-DB Rasters, a single valid query can easily exceed this limit (e.g. a dataset of 10 year's worth of rasters with one raster for each day containing minimum and maximum temperatures and we want to know the absolute min and max value for a pixel in that dataset).
+ </para>
+
+ <para>
+ The easiest change to make is the following PostgreSQL setting: <ulink url="https://www.postgresql.org/docs/current/static/runtime-config-resource.html#GUC-MAX-FILES-PER-PROCESS">max_files_per_process</ulink>. The default is set to 1000, which is far too low for Out-DB Rasters. A safe starting value could be 65536 but this really depends on your datasets and the queries run against those datasets. This setting can only be made on server start and probably only in the PostgreSQL configuration file (e.g. <emphasis>/etc/postgresql/POSTGRESQL_VERSION/CLUSTER_NAME/postgresql.conf</emphasis> in Ubuntu environments).
+
+ <programlisting>
+...
+# - Kernel Resource Usage -
+
+<emphasis role="strong">max_files_per_process = 65536</emphasis> # min 25
+ # (change requires restart)
+...
+ </programlisting>
+ </para>
+
+ <para>
+ The major change to make is the Linux kernel's open files limits. There are two parts to this:
+ <itemizedlist>
+ <listitem>
+ <para>Maximum number of open files for the entire system</para>
+ </listitem>
+ <listitem>
+ <para>Maximum number of open files per process</para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ <sect4 id="max_open_files_per_system">
+ <title>Maximum number of open files for the entire system</title>
+
+ <para>
+ You can inspect the current maximum number of open files for the entire system with the following example:
+ </para>
+
+ <programlisting>
+$ sysctl -a | grep fs.file-max
+fs.file-max = 131072
+ </programlisting>
+
+ <para>
+ If the value returned is not large enough, add a file to <emphasis>/etc/sysctl.d/</emphasis> as per the following example:
+ </para>
+
+ <programlisting>
+$ echo "fs.file-max = 6145324" >> /etc/sysctl.d/fs.conf
+
+$ cat /etc/sysctl.d/fs.conf
+fs.file-max = 6145324
+
+$ sysctl -p --system
+* Applying /etc/sysctl.d/fs.conf ...
+fs.file-max = 2097152
+* Applying /etc/sysctl.conf ...
+
+$ sysctl -a | grep fs.file-max
+fs.file-max = 6145324
+ </programlisting>
+
+ </sect4>
+ <sect4 id="max_open_files_per_process">
+ <title>Maximum number of open files per process</title>
+
+ <para>
+ We need to increase the maximum number of open files per process for the PostgreSQL server processes.
+ </para>
+
+ <para>
+ To see what the current PostgreSQL service processes are using for maximum number of open files, do as per the following example (make sure to have PostgreSQL running):
+ </para>
+
+ <programlisting>
+$ ps aux | grep postgres
+postgres 31713 0.0 0.4 179012 17564 pts/0 S Dec26 0:03 /home/dustymugs/devel/postgresql/sandbox/10/usr/local/bin/postgres -D /home/dustymugs/devel/postgresql/sandbox/10/pgdata
+postgres 31716 0.0 0.8 179776 33632 ? Ss Dec26 0:01 postgres: checkpointer process
+postgres 31717 0.0 0.2 179144 9416 ? Ss Dec26 0:05 postgres: writer process
+postgres 31718 0.0 0.2 179012 8708 ? Ss Dec26 0:06 postgres: wal writer process
+postgres 31719 0.0 0.1 179568 7252 ? Ss Dec26 0:03 postgres: autovacuum launcher process
+postgres 31720 0.0 0.1 34228 4124 ? Ss Dec26 0:09 postgres: stats collector process
+postgres 31721 0.0 0.1 179308 6052 ? Ss Dec26 0:00 postgres: bgworker: logical replication launcher
+
+$ cat /proc/31718/limits
+Limit Soft Limit Hard Limit Units
+Max cpu time unlimited unlimited seconds
+Max file size unlimited unlimited bytes
+Max data size unlimited unlimited bytes
+Max stack size 8388608 unlimited bytes
+Max core file size 0 unlimited bytes
+Max resident set unlimited unlimited bytes
+Max processes 15738 15738 processes
+<emphasis role="strong">Max open files 1024 4096 files</emphasis>
+Max locked memory 65536 65536 bytes
+Max address space unlimited unlimited bytes
+Max file locks unlimited unlimited locks
+Max pending signals 15738 15738 signals
+Max msgqueue size 819200 819200 bytes
+Max nice priority 0 0
+Max realtime priority 0 0
+Max realtime timeout unlimited unlimited us
+ </programlisting>
+
+ <para>
+ In the example above, we inspected the open files limit for Process 31718. It doesn't matter which PostgreSQL process, any of them will do. The response we are interested in is <emphasis>Max open files</emphasis>.
+ </para>
+
+ <para>
+ We want to increase <emphasis>Soft Limit</emphasis> and <emphasis>Hard Limit</emphasis> of <emphasis>Max open files</emphasis> to be greater than the value we specified for the PostgreSQL setting <varname>max_files_per_process</varname>. In our example, we set <varname>max_files_per_process</varname> to 65536.
+ </para>
+
+ <para>
+ In Ubuntu (and assuming you are using PostgreSQL's packages for Ubuntu), the easiest way to change the <emphasis>Soft Limit</emphasis> and <emphasis>Hard Limit</emphasis> is to edit <emphasis role="strong">/etc/init.d/postgresql</emphasis>. In the example, we add <emphasis role="strong">ulimit -H -n 262144</emphasis> and <emphasis role="strong">ulimit -n 131072</emphasis>.
+ </para>
+
+ <programlisting>
+...
+case "$1" in
+ start|stop|restart|reload)
+ if [ "$1" = "start" ]; then
+ create_socket_directory
+ fi
+ if [ -z "`pg_lsclusters -h`" ]; then
+ log_warning_msg 'No PostgreSQL clusters exist; see "man pg_createcluster"'
+ exit 0
+ fi
+<emphasis role="strong">
+ ulimit -H -n 262144
+ ulimit -n 131072
+</emphasis>
+ for v in $versions; do
+ $1 $v || EXIT=$?
+ done
+ exit ${EXIT:-0}
+ ;;
+ status)
+...
+</programlisting>
+
+ </sect4>
+
+ </sect3>
+
+ </sect2>
+
+ </sect1>
</chapter>
projects / postgis / commitdiff