Professional Documents
Culture Documents
Databases
=========
* :ref:`PostgreSQL <postgresql-notes>`
* :ref:`MariaDB <mariadb-notes>`
* :ref:`MySQL <mysql-notes>`
* :ref:`Oracle <oracle-notes>`
* :ref:`SQLite <sqlite-notes>`
This file describes some of the features that might be relevant to Django
usage. It is not intended as a replacement for server-specific documentation or
reference manuals.
General notes
=============
.. _persistent-database-connections:
Persistent connections
----------------------
The default value is ``0``, preserving the historical behavior of closing the
database connection at the end of each request. To enable persistent
connections, set :setting:`CONN_MAX_AGE` to a positive integer of seconds. For
unlimited persistent connections, set it to ``None``.
Connection management
~~~~~~~~~~~~~~~~~~~~~
Caveats
~~~~~~~
Since each thread maintains its own connection, your database must support at
least as many simultaneous connections as you have worker threads.
The development server creates a new thread for each request it handles,
negating the effect of persistent connections. Don't enable them during
development.
Encoding
--------
Django assumes that all databases use UTF-8 encoding. Using other encodings may
result in unexpected behavior such as "value too long" errors from your
database for data that is valid in Django. See the database specific notes
below for information on how to set up your database correctly.
.. _postgresql-notes:
PostgreSQL notes
================
.. _psycopg2: https://www.psycopg.org/
- ``client_encoding``: ``'UTF8'``,
- ``default_transaction_isolation``: ``'read committed'`` by default,
or the value set in the connection options (see below),
- ``timezone``:
- when :setting:`USE_TZ` is ``True``, ``'UTC'`` by default, or the
:setting:`TIME_ZONE <DATABASE-TIME_ZONE>` value set for the connection,
- when :setting:`USE_TZ` is ``False``, the value of the global
:setting:`TIME_ZONE` setting.
If these parameters already have the correct values, Django won't set them for
every new connection, which improves performance slightly. You can configure
them directly in :file:`postgresql.conf` or more conveniently per database
user with `ALTER ROLE`_.
Django will work just fine without this optimization, but each new connection
will do some additional queries to set these parameters.
.. _database-isolation-level:
Isolation level
---------------
import psycopg2.extensions
DATABASES = {
# ...
'OPTIONS': {
'isolation_level': psycopg2.extensions.ISOLATION_LEVEL_SERIALIZABLE,
},
}
.. note::
.. _postgresql-server-side-cursors:
Server-side cursors
-------------------
.. _cursor_tuple_fraction: https://www.postgresql.org/docs/current/runtime-config-
query.html#GUC-CURSOR-TUPLE-FRACTION
.. _transaction-pooling-server-side-cursors:
Server-side cursors are local to a connection and remain open at the end of a
transaction when :setting:`AUTOCOMMIT <DATABASE-AUTOCOMMIT>` is ``True``. A
subsequent transaction may attempt to fetch more results from a server-side
cursor. In transaction pooling mode, there's no guarantee that subsequent
transactions will use the same connection. If a different connection is used,
an error is raised when the transaction references the server-side cursor,
because server-side cursors are only accessible in the connection in which they
were created.
To benefit from server-side cursors in transaction pooling mode, you could set
up :doc:`another connection to the database </topics/db/multi-db>` in order to
perform queries that use server-side cursors. This connection needs to either
be directly to the database or to a connection pooler in session pooling mode.
.. _manually-specified-autoincrement-pk:
If you need to specify such values, reset the sequence afterwards to avoid
reusing a value that's already in the table. The :djadmin:`sqlsequencereset`
management command generates the SQL statements to do that.
.. _template: https://www.postgresql.org/docs/current/sql-createdatabase.html
.. warning::
This is dangerous: it will make your database more susceptible to data loss
or corruption in the case of a server crash or power loss. Only use this on
a development machine where you can easily restore the entire contents of
all databases in the cluster.
.. _mariadb-notes:
MariaDB notes
=============
.. versionadded:: 3.0
.. _mysql-notes:
MySQL notes
===========
Version support
---------------
Django expects the database to support Unicode (UTF-8 encoding) and delegates to
it the task of enforcing transactions and referential integrity. It is important
to be aware of the fact that the two latter ones aren't actually enforced by
MySQL when using the MyISAM storage engine, see the next section.
.. _mysql-storage-engines:
Storage engines
---------------
MySQL has several `storage engines`_. You can change the default storage engine
in the server configuration.
.. _mysql-db-api-drivers:
MySQL has a couple drivers that implement the Python Database API described in
:pep:`249`:
.. _mysqlclient: https://pypi.org/project/mysqlclient/
.. _MySQL Connector/Python: https://dev.mysql.com/downloads/connector/python
These drivers are thread-safe and provide connection pooling.
mysqlclient
~~~~~~~~~~~
MySQL Connector/Python
~~~~~~~~~~~~~~~~~~~~~~
.. _mysql-time-zone-definitions:
.. _mysql_tzinfo_to_sql: https://dev.mysql.com/doc/refman/en/mysql-tzinfo-to-
sql.html
You can `create your database`_ using the command-line tools and this SQL::
This ensures all tables and columns will use UTF-8 by default.
.. _mysql-collation:
Collation settings
~~~~~~~~~~~~~~~~~~
The collation setting for a column controls the order in which data is sorted
as well as what strings compare as equal. It can be set on a database-wide
level and also per-table and per-column. This is `documented thoroughly`_ in
the MySQL documentation. In all cases, you set the collation by directly
manipulating the database tables; Django doesn't provide a way to set this on
the model definition.
.. _documented thoroughly: https://dev.mysql.com/doc/refman/en/charset.html
Please note that according to `MySQL Unicode Character Sets`_, comparisons for
the ``utf8_general_ci`` collation are faster, but slightly less correct, than
comparisons for ``utf8_unicode_ci``. If this is acceptable for your application,
you should use ``utf8_general_ci`` because it is faster. If this is not acceptable
(for example, if you require German dictionary order), use ``utf8_unicode_ci``
because it is more accurate.
.. warning::
#. :setting:`OPTIONS`.
#. :setting:`NAME`, :setting:`USER`, :setting:`PASSWORD`, :setting:`HOST`,
:setting:`PORT`
#. MySQL option files.
# settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
# my.cnf
[client]
database = NAME
user = USER
password = PASSWORD
default-character-set = utf8
.. _mysql-sql-mode:
Setting ``sql_mode``
~~~~~~~~~~~~~~~~~~~~
From MySQL 5.7 onwards and on fresh installs of MySQL 5.6, the default value of
the ``sql_mode`` option contains ``STRICT_TRANS_TABLES``. That option escalates
warnings into errors when data are truncated upon insertion, so Django highly
recommends activating a `strict mode`_ for MySQL to prevent data loss (either
``STRICT_TRANS_TABLES`` or ``STRICT_ALL_TABLES``).
If you need to customize the SQL mode, you can set the ``sql_mode`` variable
like other MySQL options: either in a config file or with the entry
``'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"`` in the
:setting:`OPTIONS` part of your database configuration in :setting:`DATABASES`.
.. _mysql-isolation-level:
Isolation level
~~~~~~~~~~~~~~~
* ``'read uncommitted'``
* ``'read committed'``
* ``'repeatable read'``
* ``'serializable'``
If you're using a hosting service and can't change your server's default
storage engine, you have a couple of options.
'OPTIONS': {
'init_command': 'SET default_storage_engine=INNODB',
}
This sets the default storage engine upon connecting to the database.
After your tables have been created, you should remove this option as it
adds a query that is only needed during table creation to each database
connection.
Table names
-----------
There are `known issues`_ in even the latest versions of MySQL that can cause the
case of a table name to be altered when certain SQL statements are executed
under certain conditions. It is recommended that you use lowercase table
names, if possible, to avoid any problems that might arise from this behavior.
Django uses lowercase table names when it auto-generates table names from
models, so this is mainly a consideration if you are overriding the table name
via the :class:`~django.db.models.Options.db_table` parameter.
Savepoints
----------
Both the Django ORM and MySQL (when using the InnoDB :ref:`storage engine
<mysql-storage-engines>`) support database :ref:`savepoints
<topics-db-transactions-savepoints>`.
If you use the MyISAM storage engine please be aware of the fact that you will
receive database-generated errors if you try to use the :ref:`savepoint-related
methods of the transactions API <topics-db-transactions-savepoints>`. The reason
for this is that detecting the storage engine of a MySQL database/table is an
expensive operation so it was decided it isn't worth to dynamically convert
these methods in no-op's based in the results of such detection.
Character fields
~~~~~~~~~~~~~~~~
Any fields that are stored with ``VARCHAR`` column types may have their
``max_length`` restricted to 255 characters if you are using ``unique=True``
for the field. This affects :class:`~django.db.models.CharField`,
:class:`~django.db.models.SlugField`. See `the MySQL documentation`_ for more
details.
``TextField`` limitations
~~~~~~~~~~~~~~~~~~~~~~~~~
MySQL can index only the first N chars of a ``BLOB`` or ``TEXT`` column. Since
``TextField`` doesn't have a defined length, you can't mark it as
``unique=True``. MySQL will report: "BLOB/TEXT column '<db_column>' used in key
specification without a key length".
.. _mysql-fractional-seconds:
MySQL 5.6.4 and later can store fractional seconds, provided that the
column definition includes a fractional indication (e.g. ``DATETIME(6)``).
Earlier versions do not support them at all.
Django will not upgrade existing columns to include fractional seconds if the
database server supports it. If you want to enable them on an existing database,
it's up to you to either manually update the column on the target database, by
executing a command like::
``TIMESTAMP`` columns
~~~~~~~~~~~~~~~~~~~~~
If you are using a legacy database that contains ``TIMESTAMP`` columns, you must
set :setting:`USE_TZ = False <USE_TZ>` to avoid data corruption.
:djadmin:`inspectdb` maps these columns to
:class:`~django.db.models.DateTimeField` and if you enable timezone support,
both MySQL and Django will attempt to convert the values from UTC to local time.
MySQL and MariaDB do not support some options to the ``SELECT ... FOR UPDATE``
statement. If ``select_for_update()`` is used with an unsupported option, then
a :exc:`~django.db.NotSupportedError` is raised.
When performing a query on a string type, but with an integer value, MySQL will
coerce the types of all values in the table to an integer before performing the
comparison. If your table contains the values ``'abc'``, ``'def'`` and you
query for ``WHERE mycolumn=0``, both rows will match. Similarly, ``WHERE
mycolumn=1``
will match the value ``'abc1'``. Therefore, string type fields included in Django
will always cast the value to a string before using it in a query.
.. _sqlite-notes:
SQLite notes
============
.. _SQLite: https://www.sqlite.org/
.. _sqlite-string-matching:
For all SQLite versions, there is some slightly counter-intuitive behavior when
attempting to match some types of strings. These are triggered when using the
:lookup:`iexact` or :lookup:`contains` filters in Querysets. The behavior
splits into two cases:
2. For strings containing characters outside the ASCII range, all exact string
matches are performed case-sensitively, even when the case-insensitive options
are passed into the query. So the :lookup:`iexact` filter will behave exactly
the same as the :lookup:`exact` filter in these cases.
Some possible workarounds for this are `documented at sqlite.org`_, but they
aren't utilized by the default SQLite backend in Django, as incorporating them
would be fairly difficult to do robustly. Thus, Django exposes the default
SQLite behavior and you should be aware of this when doing case-insensitive or
substring filtering.
.. _sqlite-decimal-handling:
Decimal handling
----------------
SQLite has no real decimal internal type. Decimal values are internally
converted to the ``REAL`` data type (8-byte IEEE floating point number), as
explained in the `SQLite datatypes documentation`__, so they don't support
correctly-rounded decimal floating point arithmetic.
__ https://www.sqlite.org/datatype3.html#storage_classes_and_datatypes
'OPTIONS': {
# ...
'timeout': 20,
# ...
}
This will make SQLite wait a bit longer before throwing "database is locked"
errors; it won't really do anything to solve them.
``QuerySet.select_for_update()`` not supported
----------------------------------------------
SQLite does not support the ``SELECT ... FOR UPDATE`` syntax. Calling it will
have no effect.
.. _sqlite-isolation:
.. _sqlite-json1:
.. _oracle-notes:
Oracle notes
============
Django supports `Oracle Database Server`_ versions 12.2 and higher. Version
6.0 or higher of the `cx_Oracle`_ Python driver is required.
In order for the ``python manage.py migrate`` command to work, your Oracle
database user must have privileges to run the following commands:
* CREATE TABLE
* CREATE SEQUENCE
* CREATE PROCEDURE
* CREATE TRIGGER
To run a project's test suite, the user usually needs these *additional*
privileges:
* CREATE USER
* ALTER USER
* DROP USER
* CREATE TABLESPACE
* DROP TABLESPACE
* CREATE SESSION WITH ADMIN OPTION
* CREATE TABLE WITH ADMIN OPTION
* CREATE SEQUENCE WITH ADMIN OPTION
* CREATE PROCEDURE WITH ADMIN OPTION
* CREATE TRIGGER WITH ADMIN OPTION
Some test suites also create views or materialized views; to run these, the
user also needs ``CREATE VIEW WITH ADMIN OPTION`` and
``CREATE MATERIALIZED VIEW WITH ADMIN OPTION`` privileges. In particular, this
is needed for Django's own test suite.
All of these privileges are included in the DBA role, which is appropriate
for use on a private developer's database.
.. code-block:: sql
To connect using the service name of your Oracle database, your ``settings.py``
file should look something like this::
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'xe',
'USER': 'a_user',
'PASSWORD': 'a_password',
'HOST': '',
'PORT': '',
}
}
In this case, you should leave both :setting:`HOST` and :setting:`PORT` empty.
However, if you don't use a ``tnsnames.ora`` file or a similar naming method
and want to connect using the SID ("xe" in this example), then fill in both
:setting:`HOST` and :setting:`PORT` like so::
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'xe',
'USER': 'a_user',
'PASSWORD': 'a_password',
'HOST': 'dbprod01ned.mycompany.com',
'PORT': '1540',
}
}
'NAME': 'localhost:1521/orclpdb1',
'NAME': (
'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))'
'(CONNECT_DATA=(SERVICE_NAME=orclpdb1)))'
),
Threaded option
---------------
If you plan to run Django in a multithreaded environment (e.g. Apache using the
default MPM module on any modern operating system), then you **must** set
the ``threaded`` option of your Oracle database configuration to ``True``::
'OPTIONS': {
'threaded': True,
},
'OPTIONS': {
'use_returning_into': False,
},
In this case, the Oracle backend will use a separate ``SELECT`` query to
retrieve ``AutoField`` values.
Naming issues
-------------
class LegacyModel(models.Model):
class Meta:
db_table = '"name_left_in_lowercase"'
class ForeignModel(models.Model):
class Meta:
db_table = '"OTHER_USER"."NAME_ONLY_SEEMS_OVER_30"'
Quoted names can also be used with Django's other supported database
backends; except for Oracle, however, the quotes have no effect.
.. _oracle-null-empty-strings:
Django generally prefers to use the empty string (``''``) rather than
``NULL``, but Oracle treats both identically. To get around this, the
Oracle backend ignores an explicit ``null`` option on fields that
have the empty string as a possible value and generates DDL as if
``null=True``. When fetching from the database, it is assumed that
a ``NULL`` value in one of these fields really means the empty
string, and the data is silently converted to reflect this assumption.
``TextField`` limitations
-------------------------
* LOB columns may not be used in a ``SELECT DISTINCT`` list. This means that
attempting to use the ``QuerySet.distinct`` method on a model that
includes ``TextField`` columns will result in an ``ORA-00932`` error when
run against Oracle. As a workaround, use the ``QuerySet.defer`` method in
conjunction with ``distinct()`` to prevent ``TextField`` columns from being
included in the ``SELECT DISTINCT`` list.
.. _subclassing-database-backends:
Django comes with built-in database backends. You may subclass an existing
database backends to modify its behavior, features, or configuration.
Consider, for example, that you need to change a single database feature.
First, you have to create a new directory with a ``base`` module in it. For
example::
mysite/
...
mydbengine/
__init__.py
base.py
.. code-block:: python
:caption: mysite/mydbengine/base.py
class DatabaseFeatures(features.DatabaseFeatures):
def allows_group_by_selected_pks_on_model(self, model):
return True
class DatabaseWrapper(base.DatabaseWrapper):
features_class = DatabaseFeatures
DATABASES = {
'default': {
'ENGINE': 'mydbengine',
...
},
}
* `CockroachDB`_
* `Firebird`_
* `Microsoft SQL Server`_
The Django versions and ORM features supported by these unofficial backends
vary considerably. Queries regarding the specific capabilities of these
unofficial backends, along with any support queries, should be directed to
the support channels provided by each 3rd party project.
.. _CockroachDB: https://pypi.org/project/django-cockroachdb/
.. _Firebird: https://pypi.org/project/django-firebird/
.. _Microsoft SQL Server: https://pypi.org/project/django-mssql-backend/