Mapping to MongoDB

To query MongoDB data connections, you can create a mapping to them with the Mongo connector.

What is the Mongo Connector

The Mongo connector allows you to read from/write to a MongoDB database, and to execute SQL queries on Mongo collections directly from Hazelcast.

Supported SQL Statements

For the batch mode:

• SELECT
• UPDATE
• INSERT INTO/SINK INTO

For the streaming mode:

• SELECT

Installing the Connector

The Mongo Connector artifacts are published on the Maven repositories. Add the following lines to your pom.xml to include it as a dependency to your project:

xml

<dependency>
    <groupId>com.hazelcast.jet</groupId>
    <artifactId>hazelcast-jet-mongodb</artifactId>
    <version>$5.3.8</version>
</dependency>

or if you are using Gradle:

compile group: 'com.hazelcast.jet', name: 'hazelcast-jet-mongodb', version: $5.3.8.

note

To be able to use SQL over MongoDB, you have to include hazelcast-sql as well as a dependency.

Permissions

Enterprise EditionThis feature requires Hazelcast Enterprise, a licensed edition of the Hazelcast Platform.

The Mongo connector does not yet support permissions.

Before you Begin

Before you can create a mapping to a MongoDB, you must have the following:

• A $jsonSchema validation in the collection (see the schema documentation), or you have to have at least one element in the collection you want to create mapping for (for property type validation).
• Enabled operations log (oplog) if you want to use streaming mappings.

Creating a MongoDB Mapping

The following example creates a mapping to a MongoDB database.

1. In a MongoDB database, create a people collection. For example in Java, you would run the following command.

   java

   CreateCollectionOptions options = new CreateCollectionOptions();
   ValidationOptions validationOptions = new ValidationOptions();
   validationOptions.validator(BsonDocument.parse(
          "{\n" +
                  "    $jsonSchema: {\n" +
                  "      bsonType: \"object\",\n" +
                  "      title: \"Object Validation\",\n" +
                  "      properties: {" +
                  "        \"personId\": { \"bsonType\": \"int\" },\n" +
                  "        \"name\": { \"bsonType\": \"string\" }\n" +
                  "      }\n" +
                  "    }\n" +
                  "  }\n"
   ));
   options.validationOptions(validationOptions);
   database.createCollection(collectionName, options);

   The ValidationOptions are not required, but recommended.

2. Configure the data connection so that the client can be reused by multiple mappings.

   XML

   xml

   <hazelcast>
       <data-connection name="myMongo">
           <type>Mongo</type>
           <properties>
               <property name="connectionString">stringForMongo</property> (1)
           </properties>
           <shared>false</shared> (2)
       </data-connection>
   </hazelcast>

   YAML

   yaml

   data-connection:
     name: myMongo
     type: Mongo
     properties:
       connectionString: stringforMongo (1)
     shared: false (2)

   Java

   java

   DataConnectionConfig dataConnectionConfig = new DataConnectionConfig()
           .setName("myMongo")
           .setType("Mongo")
           .setProperty("connectionString", connectionStringToMongo) (1)
           .setShared(false); (2)
   config.addDataConnectionConfig(dataConnectionConfig);

   1 Your connection string.

   2 Set to true if the connection is reusable.

   Instead of providing a single connectionString parameter, you may also want to provide host, username, password and (optionally) authDb.

   Instead of providing the configuration in YAML or XML, you can also run the following SQL query.

   xml

   CREATE DATA CONNECTION myMongo type Mongo SHARED
   OPTIONS (
   	‘connectionString’ = ‘<your connection string>’
   )

3. Create the mapping.

   sql

   CREATE MAPPING people
   TYPE MONGO (1)
   DATA CONNECTION myMongo; (2)

   1 The type of the connector.

   2 The name of the data connection configuration on your members (see Step 2 above).

   In the above case, automatic schema inference will be used. You may also want to provide the schema explicitly as shown below.

   sql

   CREATE MAPPING people (
       firstName VARCHAR(100),
       lastName VARCHAR(100),
       age INT
   )
   DATA CONNECTION myMongo

   Notice that there is no mention of TYPE MONGO this time; it's automatically assumed by the SQL engine when you provide MongoDB data connection. This works with both schema provided or not.

Type Mapping

The type system in MongoDB and SQL is not exactly the same. That leads to potential confusions and the need of type coercion.

BSON Type	SQL Type	Java Type
DOUBLE	DOUBLE	DOUBLE
STRING	VARCHAR	STRING
OBJECT	OBJECT	org.bson.Document
ARRAY	OBJECT	LIST
BINDATA	-	-
UNDEFINED	-	-
OBJECTID	OBJECT	org.bson.ObjectId
BOOL	BOOLEAN	BOOLEAN
DATE This represents seconds from Unix epoch in UTC timezone. Therefore, it's not mapped to pure DATE SQL type nor LOCALDATE in Java (nor any formats with timezones).	DATE_TIME or TIMESTAMP	LOCALDATETIME
TIMESTAMP	DATE_TIME or TIMESTAMP	LOCALDATETIME
NULL	-	-
REGEX	OBJECT	org.bson.BsonRegularExpression
DBPOINTER	-	-
JAVASCRIPT	VARCHAR	STRING
JAVASCRIPTWITHSCOPE	OBJECT	org.bson.CodeWithScope
SYMBOL	-	-
INT (32 BIT)	INT	INT
LONG (64 BIT)	BIGINT	LONG
DECIMAL (128 BIT)	DECIMAL	BIGDECIMAL
MINKEY	OBJECT	org.bson.MinKey
MAXKEY	OBJECT	org.bson.MaxKey

The Java Type column represents an object returned by the SQL query if the object put into the collection is of given BSON type.

Note that, while Hazelcast is able to convert MongoDB type to the requested SQL type in the projection, the argument binding will not always work the same due to technical limitations. For example, you can have an object with the type TIMESTAMP represented as DATE_TIME, that after execution of SELECT it will give you LocalDateTime in Java client. However, binding LocalDateTime as an argument will not work, as only native MongoDB types will work for arguments. Same applies to, for example, having BSON column of type STRING mapped to INTEGER in SQL.

Type Coercion

The following table shows the possible and supported type coercions. All the default mappings from the previous section are always valid.

Type of Provided Argument	Resolved Insertion Type
LOCALDATETIME	BSONDATETIME
OFFSETDATETIME	BSONDATETIME
HazelcastJsonValue (JSON column)	DOCUMENT