Custom indexing issue
# help-with-umbraco
s
Hi, I have a custom index for our properties. Using breakpoints I can see that all of the index content is correct when it is rebuild. However, some of the properties don’t get their taxonomy channel applied, despite it being added to the “channel” field in the ValueSetBuilder.
Copy code
csharp
[Const.PropertyChannel] = string.Join(",", property.GetChannels().Select(x => ((int)x))),
this always shows as correct but for some reason anything that has more than on channel e.g. 0,1 is missing the channel field in the index. This is hard to fix as I’m not sure what isn’t right about it. The index composer is:
Copy code
csharp
builder.Services.AddExamineLuceneIndex<PropertyIndex, ConfigurationEnabledDirectoryFactory>("PropertyIndex");
builder.Services.ConfigureOptions<ConfigurePropertyIndexOptions>();
builder.Services.AddSingleton<PropertyIndexValueSetBuilder>();
builder.Services.AddSingleton<IIndexPopulator, PropertyIndexPopulator>();
builder.AddNotificationHandler<ContentCacheRefresherNotification, PropertyIndexingNotificationHandler>();
Which should be the right order as the index is working otherwise. I’m just a bit stuck now. Thanks in advance for your help! Just let me know if more info is needed and I can provide it.
l
Try not to use a comma (,), that could potentially help. When you put 1,2 in an index field, it's considered a single value, where the default parser will consider 1 2 (with a space) as two different entities. I've had this issue with a list of user id's i'd put in a custom index. Not sure if it fixes your problem, but you could try at least try 🙂
s
I'll give it a shot! Also just realised the images are missing. Basically any property where the index is above 0 and singlualr so "1" or "2" is fine but ones with multiples "1,3" or "0" doesn't get the channel field despite it appearing correct when using breakpoints to track it through visual studio
Unfortunately format doesn't seem to matter, it again all checks out but results in nothing being indexed on certain channels. It's bizarre really as it looks fine at most steps. Do you think it could be something with the property nodes themselves?
Okay so, the breakpoint for the final index value steps shows this:
Is the case that this populator is simply not complex enough or something?
Copy code
csharp
public class PropertyIndexPopulator : IndexPopulator
{
    private readonly IContentService _contentService;
    private readonly PropertyIndexValueSetBuilder _propertyIndexValueSetBuilder;

    public PropertyIndexPopulator(IContentService contentService, PropertyIndexValueSetBuilder propertyIndexValueSetBuilder)
    {
        _contentService = contentService;
        _propertyIndexValueSetBuilder = propertyIndexValueSetBuilder;
        RegisterIndex("PropertyIndex");
    }

    protected override void PopulateIndexes(IReadOnlyList<IIndex> indexes)
    {
        foreach (IIndex index in indexes)
        {
            IContent[] roots = _contentService.GetRootContent().ToArray();
            index.IndexItems(_propertyIndexValueSetBuilder.GetValueSets(roots));

            foreach (IContent root in roots)
            {
                const int pageSize = 100;
                var pageIndex = 0;
                IContent[] descendants;
                do
                {
                    descendants = _contentService.GetPagedDescendants(root.Id, pageIndex, pageSize, out _).ToArray();
                    IEnumerable<ValueSet> valueSets = _propertyIndexValueSetBuilder.GetValueSets(descendants);
                    index.IndexItems(valueSets);

                    pageIndex++;
                }
                while (descendants.Length == pageSize);
            }
        }
    }
}
I used the umbraco docs to build it all and I don't know the nuances. I also need to make other changes to the index but this one is the most important to fix as it denotes where they appear in our listings
l
What does the _propertyIndexValueSetBuilder.GetValueSets look like? Or is this a default Umbraco value set builder?
s
thanks for the response! Two secs I'll post that for you 🙂
Copy code
csharp
public IEnumerable<ValueSet> GetValueSets(params IContent[] content)
    {
        foreach (var propertyPage in content.Where(CanAddToIndex))
        {
            using var _ = _umbracoContextFactory.EnsureUmbracoContext();
            var property = _?.UmbracoContext?.Content?.GetById((int)propertyPage.Id)?.Cast<PropertyPage>();

            if (property != null)
            {
                var indexValues = PopulateValues(property);
                yield return new ValueSet(property.Id.ToString(), IndexTypes.Content, property.ContentType.Alias, indexValues);

            }
        }
    }

    private bool CanAddToIndex(IContent content) => content.ContentType.Alias == "propertyPage";

    private Dictionary<string, object> PopulateValues(PropertyPage property)
    {
        using var _ = _umbracoContextFactory.EnsureUmbracoContext();

        var values = new Dictionary<string, object>()
        {
            // Populate standard fields
            [UmbracoExamineFieldNames.NodeNameFieldName] = property.Name!,
            ["name"] = property.Name!,
            ["id"] = property.Id,
            [Const.PropertyChannel] = string.Join(",", property.GetChannels().Select(x => ((int)x))),

//code skipped here (continues adding values in this way)

return values;
}
m
Do I have a recollection that the analyzer has a bearing on what is included in the index.. and numbers are dropped as not words, for certain analyzers? Might have to set the field to a raw field to get it included?
s
Hey @Mike Chambers not sure! the thing is it is included for some properties. Just not when it has the "0" present in the string. Or if the string is multiples e.g. "0,1" as is the case in the images.
m
If it's only 1 does it drop it? (eg a number).. and "0,1" isn't a number ?? (Sorry might be a red herring)
s
If it is a single value e.g. "1" or a "2" then it is indexed correctly 🙂
m
Not sure if you have come across the SearchExtensions package.. but that has a list extension that might be of use? https://github.com/callumbwhyte/umbraco-search-extensions/blob/dev/src/Our.Umbraco.Extensions.Search/ValueTypes/ListValueType.cs though that might be once you have a list in the index 🤔
s
Yeah not too sure exactly what is not working yet. but for example here is the same unedited code producing a channel field successfully: https://cdn.discordapp.com/attachments/1346801789957046313/1347515864437755955/image.png?ex=67cc1b71&is=67cac9f1&hm=7f80338b478de91fd31fa3351b4317d26e865e36601ff0d493525dd1652b9e60&
Just never for 0s or 1,2s it's very odd
l
I've looked at some indexes I've made and usually there is a configuration where I define the types of the fields:
Copy code
csharp
options.FieldDefinitions = new FieldDefinitionCollection(
  new FieldDefinition(IndexFieldNames.Key, FieldDefinitionTypes.FullText),
  new FieldDefinition(IndexFieldNames.Id, FieldDefinitionTypes.Integer),
  new FieldDefinition(IndexFieldNames.Name, FieldDefinitionTypes.FullTextSortable),
  ...
}
Do you also have this? And is the type correct?
s
I do although as with the old website I'm converting to .NetCore from Umbraco v8 only some of the fields are defined in that list.
Copy code
csharp
options.FieldDefinitions = new(
     new FieldDefinition(Const.PropertyChannel, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.SellingPrice, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.Rent, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.Lat, FieldDefinitionTypes.Double),
     new FieldDefinition(Const.Long, FieldDefinitionTypes.Double),
     new FieldDefinition(Const.InternalAreaVal, FieldDefinitionTypes.Double),
     new FieldDefinition(Const.ExternalAreaVal, FieldDefinitionTypes.Double),
     new FieldDefinition(Const.ToLet, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.ForSale, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.SoldSTC, FieldDefinitionTypes.Integer),
     new FieldDefinition(Const.InstructionDate, FieldDefinitionTypes.Long),
     new FieldDefinition(Const.SellingInstructionDate, FieldDefinitionTypes.Long),
     new FieldDefinition(Const.LettingInstructionDate, FieldDefinitionTypes.Long),
     new FieldDefinition(Const.CreatedDate, FieldDefinitionTypes.Long),
     new FieldDefinition(Const.OfficeId, FieldDefinitionTypes.Integer)
 );
I've tried this as fulltext and fulltextsortable but neither seems to have any affect on the channel indexing
Wait damn
I think I see it
fixed
I totally blanked that the previous dev had it in at the top as integer.
l
And that's exactly why I asked 😉 Everything you said pointed to it being handled as an integer 😉
s
Thank you so much! I'd probably not have caught that myself for ages
It's amazing how no matter how long you code, it's always simple problems
l
Completely unrelated suggestion, if you include some code here on discord, you can specify the code language to get some crude coloring of the code: https://cdn.discordapp.com/attachments/1346801789957046313/1347518736998862889/image.png?ex=67cc1e1e&is=67cacc9e&hm=a6603cf8e5c3510c700356f7d8983a4822890b9a45d790a67de953dd757ad5f4&
s
Ah nice thanks, I'll do that next time! I'm sure it won't be the last time I'm caught out 🙂
l
And last suggestion, there is now a new forum: https://discord.com/channels/869656431308189746/1344602175019421736
s
I'll give it another look too 🙂
3 Views