Elasticsearch phrase suggester is suggesting me suggestions that do not exists in my index -


i have elasticsearch index have data. implemented , did-you-mean feature when user write misspelled receive suggestion right words.

i used phrase suggester because need suggestions short phrases, names example, problem suggestions not exists in index.

example:

document in index: coding master search: codning boss suggestion: <em>coding</em> boss search result: not found

my problem that, there no phrase in index match specified suggestion, it's recommending me phrases not exists , give me not found search.

what can this? shouldn't phrase suggester give suggestions phrases exists in index?

here i'll leave corresponding query, mapping , setting in case need it.

setting , mappings

{   "settings": {     "index": {       "number_of_shards": 3,       "number_of_replicas": 1,       "search.slowlog.threshold.fetch.warn": "2s",       "index.analysis.analyzer.default.filter.0": "standard",       "index.analysis.analyzer.default.tokenizer": "standard",       "index.analysis.analyzer.default.filter.1": "lowercase",       "index.analysis.analyzer.default.filter.2": "asciifolding",       "index.priority": 3,       "analysis": {         "analyzer": {           "suggests_analyzer": {             "tokenizer": "lowercase",             "filter": [               "lowercase",               "asciifolding",               "shingle_filter"             ],             "type": "custom"           }         },         "filter": {           "shingle_filter": {             "min_shingle_size": 2,             "max_shingle_size": 3,             "type": "shingle"           }         }       }     }   },   "mappings": {     "my_type": {       "properties": {         "suggest_field": {           "analyzer": "suggests_analyzer",           "type": "string"         }       }     }   } }

query

{   "didyoumean": {     "text": "codning boss",     "phrase": {       "field": "suggest_field",       "size": 1,       "gram_size": 1,       "confidence": 2.0     }   } }

thanks help.

this expected actually. if analyze document analyze api, better picture of happening. 

get suggest_index/_analyze?text=coding master&analyzer=suggests_analyzer

this output 

{    "tokens": [       {          "token": "coding",          "start_offset": 0,          "end_offset": 6,          "type": "word",          "position": 1       },       {          "token": "coding like",          "start_offset": 0,          "end_offset": 11,          "type": "shingle",          "position": 1       },       {          "token": "coding a",          "start_offset": 0,          "end_offset": 13,          "type": "shingle",          "position": 1       },       {          "token": "like",          "start_offset": 7,          "end_offset": 11,          "type": "word",          "position": 2       },       {          "token": "like a",          "start_offset": 7,          "end_offset": 13,          "type": "shingle",          "position": 2       },       {          "token": "like master",          "start_offset": 7,          "end_offset": 20,          "type": "shingle",          "position": 2       },       {          "token": "a",          "start_offset": 12,          "end_offset": 13,          "type": "word",          "position": 3       },       {          "token": "a master",          "start_offset": 12,          "end_offset": 20,          "type": "shingle",          "position": 3       },       {          "token": "master",          "start_offset": 14,          "end_offset": 20,          "type": "word",          "position": 4       }    ] }

as can see, there token "coding" generated text , hence in index. not suggesting not in index.if strictly want phrase search, might want consider using keyword tokenizer. e.g if change mapping 

{   "settings": {     "index": {       "analysis": {         "analyzer": {           "suggests_analyzer": {             "tokenizer": "lowercase",             "filter": [               "lowercase",               "asciifolding",               "shingle_filter"             ],             "type": "custom"           },           "raw_analyzer": {             "tokenizer": "keyword",             "filter": [               "lowercase",               "asciifolding"             ]           }         },         "filter": {           "shingle_filter": {             "min_shingle_size": 2,             "max_shingle_size": 3,             "type": "shingle"           }         }       }     }   },   "mappings": {     "my_type": {       "properties": {         "suggest_field": {           "analyzer": "suggests_analyzer",           "type": "string",           "fields": {             "raw": {               "analyzer": "raw_analyzer",               "type": "string"             }           }         }       }     }   } }

then query give expected results

{   "didyoumean": {     "text": "codning lke master",     "phrase": {       "field": "suggest_field.raw",       "size": 1,       "gram_size": 1     }   } }

it wont show "codning boss".

edit 1

2) comments , running phrase suggestions on own dataset, feel better approach use collate option phrase suggester provides can check every suggestion against query , give suggestion if going document index. have added stemmers mapping consider root word. using light_english less aggressive. more on that.

analyzer part of mapping looks now

 "analysis": {      "analyzer": {          "suggests_analyzer": {              "tokenizer": "standard",              "filter": [                  "lowercase",                  "english_possessive_stemmer",                  "light_english_stemmer",                  "asciifolding",                  "shingle_filter"              ],              "type": "custom"          }      },      "filter": {          "light_english_stemmer": {              "type": "stemmer",              "language": "light_english"          },          "english_possessive_stemmer": {              "type": "stemmer",              "language": "possessive_english"          },          "shingle_filter": {              "min_shingle_size": 2,              "max_shingle_size": 4,              "type": "shingle"          }      }  }

now query give desired results.

{    "suggest" : {      "text" : "appel on tabel",      "simple_phrase" : {        "phrase" : {          "field" :  "suggest_field",          "size" :   5,          "collate": {            "query": {               "inline" : {                "match_phrase": {                    "{{field_name}}" : "{{suggestion}}"                 }              }            },            "params": {"field_name" : "suggest_field"},             "prune": false          }        }      }    },    "size": 0  }

this give apple on table here match_phrase query used run every suggested phrase against index. can make "prune" : true , see results have been suggested regardless of match. might want consider using stop filter avoid stopwords.

hope helps!!


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more

ruby on rails - Seeing duplicate requests handled with Unicorn -

recently upgraded our ec2 instances c3.large in order boost more unicorn workers handle increased traffic our site. (6xworkers per machine - 2 ec2 instances). when did this, started seeing duplicate records being created! looks somehow, maybe 2 unicorn workers attempting process same request? in nginx logs see 1 [post] request particular rails controller - yet 2 records being created in database. i'm hitting kind of race condition issue - have no idea how debug this. doesn't happen time, when it's frustrating well. noticed 2 database records within 5 seconds of each other. (and yes disable button when user clicks submit button). unicorn 4.8.2 ruby 1.9.3 rails 3.2.14 any great! thanks! although may not fix problem, upgrade ruby 2.0.0. there blog mentioning idea upgrade ruby 2.0.0 if using unicorn. for example: https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app specifically: if using ruby 1.9, ...
Read more