Skip to main content

PostgreSQL Metadata Store

To use this Apache Druid extension, include postgresql-metadata-storage in the extensions load list.

Setting up PostgreSQL

To avoid issues with upgrades that require schema changes to a large metadata table, consider a PostgreSQL version that supports instant ADD COLUMN semantics.

  1. Install PostgreSQL

    Use your favorite package manager to install PostgreSQL, e.g.:

  • on Ubuntu/Debian using apt apt-get install postgresql
  • on OS X, using Homebrew brew install postgresql
  1. Create a druid database and user

    On the machine where PostgreSQL is installed, using an account with proper postgresql permissions:

    Create a druid user, enter diurd when prompted for the password.

    createuser druid -P

    Create a druid database owned by the user we just created

    createdb druid -O druid

    Note: On Ubuntu / Debian you may have to prefix the createuser and createdb commands with sudo -u postgres in order to gain proper permissions.

  2. Configure your Druid metadata storage extension:

    Add the following parameters to your Druid configuration, replacing <host> with the location (host name and port) of the database.

    druid.extensions.loadList=["postgresql-metadata-storage"]
    druid.metadata.storage.type=postgresql
    druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid
    druid.metadata.storage.connector.user=druid
    druid.metadata.storage.connector.password=diurd

Configuration

In most cases, the configuration options map directly to the postgres JDBC connection options.

PropertyDescriptionDefaultRequired
druid.metadata.postgres.ssl.useSSLEnables SSLfalseno
druid.metadata.postgres.ssl.sslPasswordThe Password Provider or String password for the client's key.noneno
druid.metadata.postgres.ssl.sslFactoryThe class name to use as the SSLSocketFactorynoneno
druid.metadata.postgres.ssl.sslFactoryArgAn optional argument passed to the sslFactory's constructornoneno
druid.metadata.postgres.ssl.sslModeThe sslMode. Possible values are "disable", "require", "verify-ca", "verify-full", "allow" and "prefer"noneno
druid.metadata.postgres.ssl.sslCertThe full path to the certificate file.noneno
druid.metadata.postgres.ssl.sslKeyThe full path to the key file.noneno
druid.metadata.postgres.ssl.sslRootCertThe full path to the root certificate.noneno
druid.metadata.postgres.ssl.sslHostNameVerifierThe classname of the hostname verifier.noneno
druid.metadata.postgres.ssl.sslPasswordCallbackThe classname of the SSL password provider.noneno
druid.metadata.postgres.dbTableSchemadruid meta table schemapublicno

PostgreSQL InputSource

The PostgreSQL extension provides an implementation of an SQL input source which can be used to ingest data into Druid from a PostgreSQL database.

{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dataSource": "some_datasource",
"dimensionsSpec": {
"dimensionExclusions": [],
"dimensions": [
"dim1",
"dim2",
"dim3"
]
},
"timestampSpec": {
"format": "auto",
"column": "ts"
},
"metricsSpec": [],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": {
"type": "none"
},
"rollup": false,
"intervals": null
},
"transformSpec": {
"filter": null,
"transforms": []
}
},
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "sql",
"database": {
"type": "postgresql",
"connectorConfig": {
"connectURI": "jdbc:postgresql://some-rds-host.us-west-1.rds.amazonaws.com:5432/druid",
"user": "admin",
"password": "secret"
}
},
"sqls": [
"SELECT * FROM some_table"
]
},
"inputFormat": {
"type": "json"
}
},
"tuningConfig": {
"type": "index_parallel"
}
}
}