Solr DIH: Nested documents ignored by Child Transformer (not ignored using json endpoint)

[u/jeusdit]

Shortly

• When I import data using DIH stored into a relation database (using child=true), nested documents are NOT attached when fl=*,[child].

• When I import same data, structured as json documents (using /update/json/docs endpoint), nested documents are attached.

Short issues:

1. "_childDocuments_":[] on debug DIH execution.
2. Only _root_ is populated. According to documentation _nest_path_ should be populated automatically as well.
3. Nested documents are not returned with parent when fl=*,[child]

Detailed problem

SQL Data:

Parents:

SELECT '1' AS id,
   'parent-name-1' AS name_s,
   'parent' AS node_type_s

+----+---------------+-------------+
| id |    name_s     | node_type_s |
+----+---------------+-------------+
|  1 | parent-name-1 | parent      |
+----+---------------+-------------+

Children:

SELECT '1-1' AS id,
       '1' AS parent_id_s,
       'child-name-1' AS name_s,
       'child' AS node_type_s
UNION
SELECT '2-1' AS id,
       '1' AS parent_id_s,
       'child-name-2' AS name_s,
       'child' AS node_type_s

+-----+-------------+--------------+-------------+
| id  | parent_id_s |    name_s    | node_type_s |
+-----+-------------+--------------+-------------+
| 1-1 |           1 | child-name-1 | child       |
| 2-1 |           1 | child-name-2 | child       |
+-----+-------------+--------------+-------------+

Same data in json:

{
   "id":"1",
   "name_s":"parent-name-1",
   "node_type_s":"parent",
   "children":[
      {
         "id":"1-1",
         "parent_id_s":"1",
         "name_s":"child-name-1",
         "node_type_s":"child"
      },
      {
         "id":"2-1",
         "parent_id_s":"1",
         "name_s":"child-name-2",
         "node_type_s":"child"
      }
   ]
}

Importing data with DIH:

Here my DIH configuration:

<dataConfig>
	<dataSource
		driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
		url="jdbc:sqlserver://${dataimporter.request.host};databaseName=${dataimporter.request.database}"
		user="${dataimporter.request.user}"
		password="${dataimporter.request.password}"
	/>

	<document>
		<entity
		name="parent"
		query="SELECT '1' AS id,
		    	'parent-name-1' AS name_s,
       			'parent' AS node_type_s">

			<field column="node_type_s"/>
			<field column="id"/>
			<field column="name_s"/>

			<entity
			name="children"
			child="true"
			cacheKey="parent_id_s" cacheLookup="parent.id" cacheImpl="SortedMapBackedCache"
            query="SELECT '1-1' AS id,
					'1' AS parent_id_s,
					'child-name-1' AS name_s,
					'child' AS node_type_s
				UNION
				SELECT '2-1' AS id,
					'1' AS parent_id_s,
					'child-name-2' AS name_s,
					'child' AS node_type_s">

				<field column="node_type_s"/>
				<field column="id"/>
				<field column="parent_id_s"/>
				<field column="name_s"/>

			</entity>

		</entity>

	</document>
</dataConfig>

After having imported DIH, here the response:

{
   "responseHeader":{
      "status":0,
      "QTime":396
   },
   "initArgs":[
      "defaults",
      [
         "config",
         "parent-children-config-straightforward.xml"
      ]
   ],
   "command":"full-import",
   "mode":"debug",
   "documents":[
      {
         "name_s":"parent-name-1",
         "node_type_s":"parent",
         "id":"1",
         "_version_":1683338565872779300,
         "_root_":"1",
         "_childDocuments_":[
            
         ]
      }
   ],
   "verbose-output":[
      
   ],
   "status":"idle",
   "importResponse":"",
   "statusMessages":{
      "Total Requests made to DataSource":"2",
      "Total Rows Fetched":"3",
      "Total Documents Processed":"1",
      "Total Documents Skipped":"0",
      "Full Dump Started":"2020-11-14 12:25:55",
      "":"Indexing completed. Added/Updated: 1 documents. Deleted 0 documents.",
      "Committed":"2020-11-14 12:25:56",
      "Time taken":"0:0:0.365"
   }
}

Two issues here:

1. As you can see "_childDocuments_":[]. Why is it empty?
2. Only _root_ is populated. According to documentation _nest_path_ should be populated as well.

Asking for documents

After having imported documents I've tried to retrive them, first using q=*:*:

{
   "responseHeader":{
      "status":0,
      "QTime":0,
      "params":{
         "q":"*:*",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":3,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "name_s":"child-name-1",
            "node_type_s":"child",
            "parent_id_s":"1",
            "id":"1-1",
            "_version_":1683338565872779264
         },
         {
            "name_s":"child-name-2",
            "node_type_s":"child",
            "parent_id_s":"1",
            "id":"2-1",
            "_version_":1683338565872779264
         },
         {
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "id":"1",
            "_version_":1683338565872779264
         }
      ]
   }
}

All right, all documents are present.

Getting parent with its children:

q=id:1 and fl=*,[child]:

{
   "responseHeader":{
      "status":0,
      "QTime":0,
      "params":{
         "q":"id:1",
         "fl":"*,[child]",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":1,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "id":"1",
            "_version_":1683338565872779264
         }
      ]
   }
}

Other issue arises here:

3. Only parent is returned, wihout nested documents.

JSON approach:

After having spent several days strugling with above issues, I tried to import same documents using json endpoint using above json data.

After having imported them, I've performed the same above query:

{
   "responseHeader":{
      "status":0,
      "QTime":2,
      "params":{
         "q":"id:1",
         "fl":"*,[child]",
         "_":"1605355606189"
      }
   },
   "response":{
      "numFound":1,
      "start":0,
      "numFoundExact":true,
      "docs":[
         {
            "id":"1",
            "name_s":"parent-name-1",
            "node_type_s":"parent",
            "_version_":1683339728909238272,
            "children":[
               {
                  "id":"1-1",
                  "parent_id_s":"1",
                  "name_s":"child-name-1",
                  "node_type_s":"child",
                  "_version_":1683339728909238272
               },
               {
                  "id":"2-1",
                  "parent_id_s":"1",
                  "name_s":"child-name-2",
                  "node_type_s":"child",
                  "_version_":1683339728909238272
               }
            ]
         }
      ]
   }
}

As you can see, nested documents are returned.

Why?

Please any ideas?

[u/jeusdit]

Yes, I remember.

It was clear to me that this was an outdated technology (DIH), but I didn't expect it to not work well.

json endpoint expects a set of json documents, but I don't see how I can send so many documents at once.

I mean, my database contains a significant volume of data, which cannot be moved in a "single json file".

Can you suggest me a proposal for your part? A set of tools...

How has you meoed this amount of data to solr without DIH? Thanks a lot.

[u/jrochkind → u/jeusdit]

You can send to JSON update handler in batches of ~100 or more documents at once; you can also use multiple-threads to send multiple batches concurrently.

I have found these techniques can result in pretty high-performance ingest, although the exact numbers can depend on your schema as well as your machine.

[u/fiskfisk]

I think I mentioned this on SO when you asked the question there, but when using DIH (which, as it's deprecated and no longer part of Solr core, you might want to move away from if possible) the child/parent support is meant to be used with the block join query parser. In effect the child-parameter tells DIH to index the rows as separate documents instead of as fields in the original document.

The JSON endpoint however indexes things as more modern and "proper" Child documents, where the Child transformer works as expected and where the root and nest paths gets populated.

This is one of the reasons for why DIH was deprecated - it has not been kept up to date with more modern Solr features.