SonicBase. In-memory embedded or distributed streaming sql database

Elasticsearch Integration

SonicBase includes a streams provider that allows you to stream inserts, updates and deletes to your Elasticsearch cluster.

Configuration

To integrate with Elasticsearch you must configure the streams provider in your cluster config file. An example configuration follows:

streams:
  processorThreadCount: 8
  producers:
  - producer:
      className: com.sonicbase.streams.ElasticsearchProducer
      mappingFile: es-mapping.yaml
      maxBatchSize: 200
The mapping file must be located in the $SONICBASE_HOME/config directory.

Mapping File

You must provide a mapping file to instruct SonicBase on how to send the data to ElasticSearch. An example mapping file is shown below

esClusters:
- esCluster:
  name: es1
  hosts: 127.0.0.1:9200
sbClusters:
- sbCluster:
  name: 1-local
  databases:
  - database:
    name: db
    tables:
    - table:
      name: persons
      esCluster: es1
      esIndex: persons
      fields:
        id1: id1
        id2: id2
        name: name
        socialsecuritynumber: ssn

You provide settings for Elasticsearch clusters, SonicBase clusters, databases and tables.
  • esCluster - name and comma delimited list of hosts for the cluster
  • sbCluster - name of the SonicBase cluster and list of databases in the cluster you want to integrate
  • database - name of the SonicBase database and list of tables in the database you want to integrate
  • table - name of the SonicBase table, fields and the esCluster and esIndex you want to send records to
  • fields - Sonicbase field name, followed by a colon, followed by the esIndex field name