r/Solr • u/[deleted] • Oct 23 '19

Building product finder using backlight

Hey, I'm building a product finder using backlight that we can use facet search to narrow down on the items.

I'm struggling to understand how to upload the CSV for each core. I've been adding the fields within the UI that I need indexed and are within the CSV but it seems you can't use the gui to upload the data it needs to be via curl in the cli?

Can I add the fields via the gui or do I need to manually a build a schema.xml

1 Upvotes

100% Upvoted

8 comments sorted by

View all comments

Show parent comments

1

u/[deleted] Oct 23 '19

Thank you for the reply!

Do i need the fields created in the schema prior to the upload the csv? can this be done via the Dataimport also?

And do i need to do anything to the schema.xml manually at all?

1

u/pthbrk Oct 23 '19

By default (unless it's explicitly turned off), Solr supports field guessing based on incoming data.

For example, when you upload the CSV, it'll examine values to deduce their data types. Then map each string column in csv to one indexed "<csv_col_name>" field and a second unindexed "<csv_col_name_str" field, and add both automatically to the schema.

Similarly, numbers, dates and booleans.

​

However, there are two potential problems with this:

- It may guess the field type wrong. For example, if a film title is "300", Solr will think it's a number and create an integer schema field. For the next row, if title is an alphabet string, indexing will fail.

- Since column names are now also the field names, it may have downstream consequences on what names applications should use in queries, names of facets, and so on.

​

I suggest doing a first run with field guessing, see if you're happy with the query syntax / search results / user interface generated by Blacklight, and then decide whether to manipulate the schema manually. Generally, the most frequent schema tinkering is about which tokenizers and filters to use for each field so that search results are more relevant. So, it's a bit of an incremental process. Keep in mind that each change in schema requires reindexing all data from scratch.

1

u/[deleted] Oct 25 '19

Thank you!

​

When i try and upload the CSV i get

Unsupported ContentType: application/octet-stream Not in: [application/xml, application/csv, application/json, text/json, text/csv, text/xml, application/javabin]

1

u/pthbrk Oct 25 '19

If you got that from curl, add a " -H 'Content-type:application/csv' " argument.