PostgreSQL Metadata Store
To use this Apache Druid extension, include postgresql-metadata-storage
in the extensions load list.
Setting up PostgreSQL
To avoid issues with upgrades that require schema changes to a large metadata table, consider a PostgreSQL version that supports instant ADD COLUMN semantics.
Install PostgreSQL
Use your favorite package manager to install PostgreSQL, e.g.:
- on Ubuntu/Debian using apt
apt-get install postgresql
- on OS X, using Homebrew
brew install postgresql
Create a druid database and user
On the machine where PostgreSQL is installed, using an account with proper postgresql permissions:
Create a druid user, enter
diurd
when prompted for the password.createuser druid -P
Create a druid database owned by the user we just created
createdb druid -O druid
Note: On Ubuntu / Debian you may have to prefix the
createuser
andcreatedb
commands withsudo -u postgres
in order to gain proper permissions.Configure your Druid metadata storage extension:
Add the following parameters to your Druid configuration, replacing
<host>
with the location (host name and port) of the database.druid.extensions.loadList=["postgresql-metadata-storage"]
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd
Configuration
In most cases, the configuration options map directly to the postgres JDBC connection options.
Property | Description | Default | Required |
---|---|---|---|
druid.metadata.postgres.ssl.useSSL | Enables SSL | false | no |
druid.metadata.postgres.ssl.sslPassword | The Password Provider or String password for the client's key. | none | no |
druid.metadata.postgres.ssl.sslFactory | The class name to use as the SSLSocketFactory | none | no |
druid.metadata.postgres.ssl.sslFactoryArg | An optional argument passed to the sslFactory's constructor | none | no |
druid.metadata.postgres.ssl.sslMode | The sslMode. Possible values are "disable", "require", "verify-ca", "verify-full", "allow" and "prefer" | none | no |
druid.metadata.postgres.ssl.sslCert | The full path to the certificate file. | none | no |
druid.metadata.postgres.ssl.sslKey | The full path to the key file. | none | no |
druid.metadata.postgres.ssl.sslRootCert | The full path to the root certificate. | none | no |
druid.metadata.postgres.ssl.sslHostNameVerifier | The classname of the hostname verifier. | none | no |
druid.metadata.postgres.ssl.sslPasswordCallback | The classname of the SSL password provider. | none | no |
druid.metadata.postgres.dbTableSchema | druid meta table schema | public | no |
PostgreSQL InputSource
The PostgreSQL extension provides an implementation of an SQL input source which can be used to ingest data into Druid from a PostgreSQL database.
{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dataSource": "some_datasource",
"dimensionsSpec": {
"dimensionExclusions": [],
"dimensions": [
"dim1",
"dim2",
"dim3"
]
},
"timestampSpec": {
"format": "auto",
"column": "ts"
},
"metricsSpec": [],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "DAY",
"queryGranularity": {
"type": "none"
},
"rollup": false,
"intervals": null
},
"transformSpec": {
"filter": null,
"transforms": []
}
},
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "sql",
"database": {
"type": "postgresql",
"connectorConfig": {
"connectURI": "jdbc:postgresql://some-rds-host.us-west-1.rds.amazonaws.com:5432/druid",
"user": "admin",
"password": "secret"
}
},
"sqls": [
"SELECT * FROM some_table"
]
},
"inputFormat": {
"type": "json"
}
},
"tuningConfig": {
"type": "index_parallel"
}
}
}