mirror of https://github.com/go-gitea/gitea.git
Issue search support elasticsearch (#9428)
* Issue search support elasticsearch * Fix lint * Add indexer name on app.ini * add a warnning on SearchIssuesByKeyword * improve codepull/10263/head
Lunny Xiao
GitHub
parent
17656021f1
commit
5dbf36f356
|
|
@ -86,6 +86,12 @@ services:
|
|||
pull: default
|
||||
image: gitea/test-openldap:latest
|
||||
|
||||
- name: elasticsearch
|
||||
pull: default
|
||||
environment:
|
||||
discovery.type: single-node
|
||||
image: elasticsearch:7.5.0
|
||||
|
||||
steps:
|
||||
- name: fetch-tags
|
||||
pull: default
|
||||
|
|
|
|||
|
|
@ -368,8 +368,12 @@ CONN_MAX_LIFETIME = 3s
|
|||
MAX_OPEN_CONNS = 0
|
||||
|
||||
[indexer]
|
||||
; Issue indexer type, currently support: bleve or db, default is bleve
|
||||
; Issue indexer type, currently support: bleve, db or elasticsearch, default is bleve
|
||||
ISSUE_INDEXER_TYPE = bleve
|
||||
; Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch
|
||||
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@localhost:9200
|
||||
; Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
|
||||
ISSUE_INDEXER_NAME = gitea_issues
|
||||
; Issue indexer storage path, available when ISSUE_INDEXER_TYPE is bleve
|
||||
ISSUE_INDEXER_PATH = indexers/issues.bleve
|
||||
; Issue indexer queue, currently support: channel, levelqueue or redis, default is levelqueue
|
||||
|
|
|
|||
|
|
@ -228,8 +228,10 @@ relation to port exhaustion.
|
|||
|
||||
## Indexer (`indexer`)
|
||||
|
||||
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently support: bleve or db, if it's db, below issue indexer item will be invalid.
|
||||
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search.
|
||||
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently supported: `bleve`, `db` or `elasticsearch`.
|
||||
- `ISSUE_INDEXER_CONN_STR`: ****: Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch. i.e. http://elastic:changeme@localhost:9200
|
||||
- `ISSUE_INDEXER_NAME`: **gitea_issues**: Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
|
||||
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search; available when ISSUE_INDEXER_TYPE is bleve and elasticsearch.
|
||||
- The next 4 configuration values are deprecated and should be set in `queue.issue_indexer` however are kept for backwards compatibility:
|
||||
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: Issue indexer queue, currently supports:`channel`, `levelqueue`, `redis`.
|
||||
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: When `ISSUE_INDEXER_QUEUE_TYPE` is `levelqueue`, this will be the queue will be saved path.
|
||||
|
|
|
|||
|
|
@ -89,7 +89,9 @@ menu:
|
|||
|
||||
## Indexer (`indexer`)
|
||||
|
||||
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型，当前支持 `bleve` 或 `db`，当为 `db` 时其它工单索引项可不用设置。
|
||||
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型，当前支持 `bleve`, `db` 和 `elasticsearch`，当为 `db` 时其它工单索引项可不用设置。
|
||||
- `ISSUE_INDEXER_CONN_STR`: ****: 工单索引连接字符串，仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。例如: http://elastic:changeme@localhost:9200
|
||||
- `ISSUE_INDEXER_NAME`: **gitea_issues**: 工单索引名称，仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。
|
||||
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: 工单索引文件存放路径，当索引类型为 `bleve` 时有效。
|
||||
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: 工单索引队列类型，当前支持 `channel`， `levelqueue` 或 `redis`。
|
||||
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: 当 `ISSUE_INDEXER_QUEUE_TYPE` 为 `levelqueue` 时，保存索引队列的磁盘路径。
|
||||
|
|
|
|||
1
go.mod
1
go.mod
|
|
@ -74,6 +74,7 @@ require (
|
|||
github.com/nfnt/resize v0.0.0-20160724205520-891127d8d1b5
|
||||
github.com/niklasfasching/go-org v0.1.9
|
||||
github.com/oliamb/cutter v0.2.2
|
||||
github.com/olivere/elastic/v7 v7.0.9
|
||||
github.com/pkg/errors v0.8.1
|
||||
github.com/pquerna/otp v0.0.0-20160912161815-54653902c20e
|
||||
github.com/prometheus/client_golang v1.1.0
|
||||
|
|
|
|||
10
go.sum
10
go.sum
|
|
@ -68,6 +68,7 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
|
|||
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
|
||||
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
|
||||
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
|
||||
github.com/aws/aws-sdk-go v1.25.25/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
|
||||
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
|
||||
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
|
|
@ -154,6 +155,8 @@ github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 h1:E2s37DuLxFhQD
|
|||
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
|
||||
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
|
||||
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
|
||||
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
|
||||
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
|
||||
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
|
|
@ -248,6 +251,7 @@ github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZ
|
|||
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
|
||||
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
|
||||
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
|
||||
|
|
@ -322,6 +326,7 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOl
|
|||
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
|
||||
github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA=
|
||||
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
|
||||
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
|
||||
github.com/jmhodges/levigo v1.0.0 h1:q5EC36kV79HWeTBWsod3mG11EgStG3qArTKcvlksN1U=
|
||||
github.com/jmhodges/levigo v1.0.0/go.mod h1:Q6Qx+uH3RAqyK4rFQroq9RL7mdkABMcfhEI+nNuzMJQ=
|
||||
github.com/joho/godotenv v1.3.0 h1:Zjp+RcGpHhGlrMbJzXTrZZPrWj+1vfm90La1wgB6Bhc=
|
||||
|
|
@ -411,6 +416,8 @@ github.com/niklasfasching/go-org v0.1.9/go.mod h1:AsLD6X7djzRIz4/RFZu8vwRL0VGjUv
|
|||
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
|
||||
github.com/oliamb/cutter v0.2.2 h1:Lfwkya0HHNU1YLnGv2hTkzHfasrSMkgv4Dn+5rmlk3k=
|
||||
github.com/oliamb/cutter v0.2.2/go.mod h1:4BenG2/4GuRBDbVm/OPahDVqbrOemzpPiG5mi1iryBU=
|
||||
github.com/olivere/elastic/v7 v7.0.9 h1:+bTR1xJbfLYD8WnTBt9672mFlKxjfWRJpEQ1y8BMS3g=
|
||||
github.com/olivere/elastic/v7 v7.0.9/go.mod h1:2TeRd0vhLRTK9zqm5xP0uLiVeZ5yUoL7kZ+8SZA9r9Y=
|
||||
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
|
||||
|
|
@ -418,6 +425,7 @@ github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
|
|||
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
|
||||
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
|
||||
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
|
||||
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
|
||||
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
|
||||
|
|
@ -486,6 +494,7 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
|
|||
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||
github.com/smartystreets/assertions v1.0.1 h1:voD4ITNjPL5jjBfgR/r8fPIIBrliWrWHeiJApdr3r4w=
|
||||
github.com/smartystreets/assertions v1.0.1/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
|
||||
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
|
||||
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
|
||||
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||
github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8=
|
||||
|
|
@ -573,6 +582,7 @@ go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qL
|
|||
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
|
||||
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
|
||||
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
|
||||
go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA=
|
||||
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
|
||||
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
|
||||
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@ PASSWD = {{TEST_MYSQL_PASSWORD}}
|
|||
SSL_MODE = disable
|
||||
|
||||
[indexer]
|
||||
ISSUE_INDEXER_TYPE = elasticsearch
|
||||
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@elasticsearch:9200
|
||||
ISSUE_INDEXER_PATH = integrations/indexers-mysql/issues.bleve
|
||||
REPO_INDEXER_ENABLED = true
|
||||
REPO_INDEXER_PATH = integrations/indexers-mysql/repos.bleve
|
||||
|
|
|
|||
|
|
@ -170,7 +170,7 @@ func NewBleveIndexer(indexDir string) *BleveIndexer {
|
|||
}
|
||||
}
|
||||
|
||||
// Init will initial the indexer
|
||||
// Init will initialize the indexer
|
||||
func (b *BleveIndexer) Init() (bool, error) {
|
||||
var err error
|
||||
b.indexer, err = openIndexer(b.indexDir, issueIndexerLatestVersion)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,230 @@
|
|||
// Copyright 2019 The Gitea Authors. All rights reserved.
|
||||
// Use of this source code is governed by a MIT-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
package issues
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"code.gitea.io/gitea/modules/log"
|
||||
|
||||
"github.com/olivere/elastic/v7"
|
||||
)
|
||||
|
||||
var (
|
||||
_ Indexer = &ElasticSearchIndexer{}
|
||||
)
|
||||
|
||||
// ElasticSearchIndexer implements Indexer interface
|
||||
type ElasticSearchIndexer struct {
|
||||
client *elastic.Client
|
||||
indexerName string
|
||||
}
|
||||
|
||||
type elasticLogger struct {
|
||||
*log.Logger
|
||||
}
|
||||
|
||||
func (l elasticLogger) Printf(format string, args ...interface{}) {
|
||||
_ = l.Logger.Log(2, l.Logger.GetLevel(), format, args...)
|
||||
}
|
||||
|
||||
// NewElasticSearchIndexer creates a new elasticsearch indexer
|
||||
func NewElasticSearchIndexer(url, indexerName string) (*ElasticSearchIndexer, error) {
|
||||
opts := []elastic.ClientOptionFunc{
|
||||
elastic.SetURL(url),
|
||||
elastic.SetSniff(false),
|
||||
elastic.SetHealthcheckInterval(10 * time.Second),
|
||||
elastic.SetGzip(false),
|
||||
}
|
||||
|
||||
logger := elasticLogger{log.GetLogger(log.DEFAULT)}
|
||||
|
||||
if logger.GetLevel() == log.TRACE || logger.GetLevel() == log.DEBUG {
|
||||
opts = append(opts, elastic.SetTraceLog(logger))
|
||||
} else if logger.GetLevel() == log.ERROR || logger.GetLevel() == log.CRITICAL || logger.GetLevel() == log.FATAL {
|
||||
opts = append(opts, elastic.SetErrorLog(logger))
|
||||
} else if logger.GetLevel() == log.INFO || logger.GetLevel() == log.WARN {
|
||||
opts = append(opts, elastic.SetInfoLog(logger))
|
||||
}
|
||||
|
||||
client, err := elastic.NewClient(opts...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &ElasticSearchIndexer{
|
||||
client: client,
|
||||
indexerName: indexerName,
|
||||
}, nil
|
||||
}
|
||||
|
||||
const (
|
||||
defaultMapping = `{
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer",
|
||||
"index": true
|
||||
},
|
||||
"repo_id": {
|
||||
"type": "integer",
|
||||
"index": true
|
||||
},
|
||||
"title": {
|
||||
"type": "text",
|
||||
"index": true
|
||||
},
|
||||
"content": {
|
||||
"type": "text",
|
||||
"index": true
|
||||
},
|
||||
"comments": {
|
||||
"type" : "text",
|
||||
"index": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
)
|
||||
|
||||
// Init will initialize the indexer
|
||||
func (b *ElasticSearchIndexer) Init() (bool, error) {
|
||||
ctx := context.Background()
|
||||
exists, err := b.client.IndexExists(b.indexerName).Do(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
if !exists {
|
||||
var mapping = defaultMapping
|
||||
|
||||
createIndex, err := b.client.CreateIndex(b.indexerName).BodyString(mapping).Do(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !createIndex.Acknowledged {
|
||||
return false, errors.New("init failed")
|
||||
}
|
||||
|
||||
return false, nil
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Index will save the index data
|
||||
func (b *ElasticSearchIndexer) Index(issues []*IndexerData) error {
|
||||
if len(issues) == 0 {
|
||||
return nil
|
||||
} else if len(issues) == 1 {
|
||||
issue := issues[0]
|
||||
_, err := b.client.Index().
|
||||
Index(b.indexerName).
|
||||
Id(fmt.Sprintf("%d", issue.ID)).
|
||||
BodyJson(map[string]interface{}{
|
||||
"id": issue.ID,
|
||||
"repo_id": issue.RepoID,
|
||||
"title": issue.Title,
|
||||
"content": issue.Content,
|
||||
"comments": issue.Comments,
|
||||
}).
|
||||
Do(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
reqs := make([]elastic.BulkableRequest, 0)
|
||||
for _, issue := range issues {
|
||||
reqs = append(reqs,
|
||||
elastic.NewBulkIndexRequest().
|
||||
Index(b.indexerName).
|
||||
Id(fmt.Sprintf("%d", issue.ID)).
|
||||
Doc(map[string]interface{}{
|
||||
"id": issue.ID,
|
||||
"repo_id": issue.RepoID,
|
||||
"title": issue.Title,
|
||||
"content": issue.Content,
|
||||
"comments": issue.Comments,
|
||||
}),
|
||||
)
|
||||
}
|
||||
|
||||
_, err := b.client.Bulk().
|
||||
Index(b.indexerName).
|
||||
Add(reqs...).
|
||||
Do(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
// Delete deletes indexes by ids
|
||||
func (b *ElasticSearchIndexer) Delete(ids ...int64) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
} else if len(ids) == 1 {
|
||||
_, err := b.client.Delete().
|
||||
Index(b.indexerName).
|
||||
Id(fmt.Sprintf("%d", ids[0])).
|
||||
Do(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
reqs := make([]elastic.BulkableRequest, 0)
|
||||
for _, id := range ids {
|
||||
reqs = append(reqs,
|
||||
elastic.NewBulkDeleteRequest().
|
||||
Index(b.indexerName).
|
||||
Id(fmt.Sprintf("%d", id)),
|
||||
)
|
||||
}
|
||||
|
||||
_, err := b.client.Bulk().
|
||||
Index(b.indexerName).
|
||||
Add(reqs...).
|
||||
Do(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
// Search searches for issues by given conditions.
|
||||
// Returns the matching issue IDs
|
||||
func (b *ElasticSearchIndexer) Search(keyword string, repoIDs []int64, limit, start int) (*SearchResult, error) {
|
||||
kwQuery := elastic.NewMultiMatchQuery(keyword, "title", "content", "comments")
|
||||
query := elastic.NewBoolQuery()
|
||||
query = query.Must(kwQuery)
|
||||
if len(repoIDs) > 0 {
|
||||
var repoStrs = make([]interface{}, 0, len(repoIDs))
|
||||
for _, repoID := range repoIDs {
|
||||
repoStrs = append(repoStrs, repoID)
|
||||
}
|
||||
repoQuery := elastic.NewTermsQuery("repo_id", repoStrs...)
|
||||
query = query.Must(repoQuery)
|
||||
}
|
||||
searchResult, err := b.client.Search().
|
||||
Index(b.indexerName).
|
||||
Query(query).
|
||||
Sort("id", true).
|
||||
From(start).Size(limit).
|
||||
Do(context.Background())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
hits := make([]Match, 0, limit)
|
||||
for _, hit := range searchResult.Hits.Hits {
|
||||
id, _ := strconv.ParseInt(hit.Id, 10, 64)
|
||||
hits = append(hits, Match{
|
||||
ID: id,
|
||||
})
|
||||
}
|
||||
|
||||
return &SearchResult{
|
||||
Total: searchResult.TotalHits(),
|
||||
Hits: hits,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Close implements indexer
|
||||
func (b *ElasticSearchIndexer) Close() {}
|
||||
|
|
@ -21,13 +21,13 @@ import (
|
|||
|
||||
// IndexerData data stored in the issue indexer
|
||||
type IndexerData struct {
|
||||
ID int64
|
||||
RepoID int64
|
||||
Title string
|
||||
Content string
|
||||
Comments []string
|
||||
IsDelete bool
|
||||
IDs []int64
|
||||
ID int64 `json:"id"`
|
||||
RepoID int64 `json:"repo_id"`
|
||||
Title string `json:"title"`
|
||||
Content string `json:"content"`
|
||||
Comments []string `json:"comments"`
|
||||
IsDelete bool `json:"is_delete"`
|
||||
IDs []int64 `json:"ids"`
|
||||
}
|
||||
|
||||
// Match represents on search result
|
||||
|
|
@ -100,7 +100,7 @@ func InitIssueIndexer(syncReindex bool) {
|
|||
|
||||
// Create the Queue
|
||||
switch setting.Indexer.IssueType {
|
||||
case "bleve":
|
||||
case "bleve", "elasticsearch":
|
||||
handler := func(data ...queue.Data) {
|
||||
indexer := holder.get()
|
||||
if indexer == nil {
|
||||
|
|
@ -160,6 +160,19 @@ func InitIssueIndexer(syncReindex bool) {
|
|||
log.Info("PID: %d Issue Indexer closed", os.Getpid())
|
||||
})
|
||||
log.Debug("Created Bleve Indexer")
|
||||
case "elasticsearch":
|
||||
graceful.GetManager().RunWithShutdownFns(func(_, atTerminate func(context.Context, func())) {
|
||||
issueIndexer, err := NewElasticSearchIndexer(setting.Indexer.IssueConnStr, "gitea_issues")
|
||||
if err != nil {
|
||||
log.Fatal("Unable to initialize Elastic Search Issue Indexer: %v", err)
|
||||
}
|
||||
exist, err := issueIndexer.Init()
|
||||
if err != nil {
|
||||
log.Fatal("Unable to issueIndexer.Init: %v", err)
|
||||
}
|
||||
populate = !exist
|
||||
holder.set(issueIndexer)
|
||||
})
|
||||
case "db":
|
||||
issueIndexer := &DBIndexer{}
|
||||
holder.set(issueIndexer)
|
||||
|
|
@ -308,6 +321,7 @@ func DeleteRepoIssueIndexer(repo *models.Repository) {
|
|||
}
|
||||
|
||||
// SearchIssuesByKeyword search issue ids by keywords and repo id
|
||||
// WARNNING: You have to ensure user have permission to visit repoIDs' issues
|
||||
func SearchIssuesByKeyword(repoIDs []int64, keyword string) ([]int64, error) {
|
||||
var issueIDs []int64
|
||||
indexer := holder.get()
|
||||
|
|
@ -316,7 +330,7 @@ func SearchIssuesByKeyword(repoIDs []int64, keyword string) ([]int64, error) {
|
|||
log.Error("SearchIssuesByKeyword(): unable to get indexer!")
|
||||
return nil, fmt.Errorf("unable to get issue indexer")
|
||||
}
|
||||
res, err := indexer.Search(keyword, repoIDs, 1000, 0)
|
||||
res, err := indexer.Search(keyword, repoIDs, 50, 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,20 +27,25 @@ var (
|
|||
Indexer = struct {
|
||||
IssueType string
|
||||
IssuePath string
|
||||
RepoIndexerEnabled bool
|
||||
RepoPath string
|
||||
UpdateQueueLength int
|
||||
MaxIndexerFileSize int64
|
||||
IssueConnStr string
|
||||
IssueIndexerName string
|
||||
IssueQueueType string
|
||||
IssueQueueDir string
|
||||
IssueQueueConnStr string
|
||||
IssueQueueBatchNumber int
|
||||
StartupTimeout time.Duration
|
||||
IncludePatterns []glob.Glob
|
||||
ExcludePatterns []glob.Glob
|
||||
|
||||
RepoIndexerEnabled bool
|
||||
RepoPath string
|
||||
UpdateQueueLength int
|
||||
MaxIndexerFileSize int64
|
||||
IncludePatterns []glob.Glob
|
||||
ExcludePatterns []glob.Glob
|
||||
}{
|
||||
IssueType: "bleve",
|
||||
IssuePath: "indexers/issues.bleve",
|
||||
IssueConnStr: "",
|
||||
IssueIndexerName: "gitea_issues",
|
||||
IssueQueueType: LevelQueueType,
|
||||
IssueQueueDir: "indexers/issues.queue",
|
||||
IssueQueueConnStr: "",
|
||||
|
|
@ -57,6 +62,14 @@ func newIndexerService() {
|
|||
if !filepath.IsAbs(Indexer.IssuePath) {
|
||||
Indexer.IssuePath = path.Join(AppWorkPath, Indexer.IssuePath)
|
||||
}
|
||||
Indexer.IssueConnStr = sec.Key("ISSUE_INDEXER_CONN_STR").MustString(Indexer.IssueConnStr)
|
||||
Indexer.IssueIndexerName = sec.Key("ISSUE_INDEXER_NAME").MustString(Indexer.IssueIndexerName)
|
||||
|
||||
Indexer.IssueQueueType = sec.Key("ISSUE_INDEXER_QUEUE_TYPE").MustString(LevelQueueType)
|
||||
Indexer.IssueQueueDir = sec.Key("ISSUE_INDEXER_QUEUE_DIR").MustString(path.Join(AppDataPath, "indexers/issues.queue"))
|
||||
Indexer.IssueQueueConnStr = sec.Key("ISSUE_INDEXER_QUEUE_CONN_STR").MustString(path.Join(AppDataPath, ""))
|
||||
Indexer.IssueQueueBatchNumber = sec.Key("ISSUE_INDEXER_QUEUE_BATCH_NUMBER").MustInt(20)
|
||||
|
||||
Indexer.RepoIndexerEnabled = sec.Key("REPO_INDEXER_ENABLED").MustBool(false)
|
||||
Indexer.RepoPath = sec.Key("REPO_INDEXER_PATH").MustString(path.Join(AppDataPath, "indexers/repos.bleve"))
|
||||
if !filepath.IsAbs(Indexer.RepoPath) {
|
||||
|
|
@ -64,13 +77,8 @@ func newIndexerService() {
|
|||
}
|
||||
Indexer.IncludePatterns = IndexerGlobFromString(sec.Key("REPO_INDEXER_INCLUDE").MustString(""))
|
||||
Indexer.ExcludePatterns = IndexerGlobFromString(sec.Key("REPO_INDEXER_EXCLUDE").MustString(""))
|
||||
|
||||
Indexer.UpdateQueueLength = sec.Key("UPDATE_BUFFER_LEN").MustInt(20)
|
||||
Indexer.MaxIndexerFileSize = sec.Key("MAX_FILE_SIZE").MustInt64(1024 * 1024)
|
||||
Indexer.IssueQueueType = sec.Key("ISSUE_INDEXER_QUEUE_TYPE").MustString(LevelQueueType)
|
||||
Indexer.IssueQueueDir = sec.Key("ISSUE_INDEXER_QUEUE_DIR").MustString(path.Join(AppDataPath, "indexers/issues.queue"))
|
||||
Indexer.IssueQueueConnStr = sec.Key("ISSUE_INDEXER_QUEUE_CONN_STR").MustString(path.Join(AppDataPath, ""))
|
||||
Indexer.IssueQueueBatchNumber = sec.Key("ISSUE_INDEXER_QUEUE_BATCH_NUMBER").MustInt(20)
|
||||
Indexer.StartupTimeout = sec.Key("STARTUP_TIMEOUT").MustDuration(30 * time.Second)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
.root
|
||||
*_easyjson.go
|
||||
*.iml
|
||||
.idea
|
||||
*.swp
|
||||
|
|
@ -0,0 +1,12 @@
|
|||
language: go
|
||||
|
||||
go:
|
||||
- tip
|
||||
- stable
|
||||
|
||||
matrix:
|
||||
allow_failures:
|
||||
- go: tip
|
||||
|
||||
install:
|
||||
- go get golang.org/x/lint/golint
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
all: test
|
||||
|
||||
clean:
|
||||
rm -rf bin
|
||||
rm -rf tests/*_easyjson.go
|
||||
rm -rf benchmark/*_easyjson.go
|
||||
|
||||
build:
|
||||
go build -i -o ./bin/easyjson ./easyjson
|
||||
|
||||
generate: build
|
||||
bin/easyjson -stubs \
|
||||
./tests/snake.go \
|
||||
./tests/data.go \
|
||||
./tests/omitempty.go \
|
||||
./tests/nothing.go \
|
||||
./tests/named_type.go \
|
||||
./tests/custom_map_key_type.go \
|
||||
./tests/embedded_type.go \
|
||||
./tests/reference_to_pointer.go \
|
||||
|
||||
bin/easyjson -all ./tests/data.go
|
||||
bin/easyjson -all ./tests/nothing.go
|
||||
bin/easyjson -all ./tests/errors.go
|
||||
bin/easyjson -snake_case ./tests/snake.go
|
||||
bin/easyjson -omit_empty ./tests/omitempty.go
|
||||
bin/easyjson -build_tags=use_easyjson ./benchmark/data.go
|
||||
bin/easyjson ./tests/nested_easy.go
|
||||
bin/easyjson ./tests/named_type.go
|
||||
bin/easyjson ./tests/custom_map_key_type.go
|
||||
bin/easyjson ./tests/embedded_type.go
|
||||
bin/easyjson ./tests/reference_to_pointer.go
|
||||
bin/easyjson ./tests/key_marshaler_map.go
|
||||
bin/easyjson -disallow_unknown_fields ./tests/disallow_unknown.go
|
||||
|
||||
test: generate
|
||||
go test \
|
||||
./tests \
|
||||
./jlexer \
|
||||
./gen \
|
||||
./buffer
|
||||
cd benchmark && go test -benchmem -tags use_easyjson -bench .
|
||||
golint -set_exit_status ./tests/*_easyjson.go
|
||||
|
||||
bench-other: generate
|
||||
cd benchmark && make
|
||||
|
||||
bench-python:
|
||||
benchmark/ujson.sh
|
||||
|
||||
|
||||
.PHONY: clean generate test build
|
||||
|
|
@ -0,0 +1,336 @@
|
|||
# easyjson [](https://travis-ci.org/mailru/easyjson) [](https://goreportcard.com/report/github.com/mailru/easyjson)
|
||||
|
||||
Package easyjson provides a fast and easy way to marshal/unmarshal Go structs
|
||||
to/from JSON without the use of reflection. In performance tests, easyjson
|
||||
outperforms the standard `encoding/json` package by a factor of 4-5x, and other
|
||||
JSON encoding packages by a factor of 2-3x.
|
||||
|
||||
easyjson aims to keep generated Go code simple enough so that it can be easily
|
||||
optimized or fixed. Another goal is to provide users with the ability to
|
||||
customize the generated code by providing options not available with the
|
||||
standard `encoding/json` package, such as generating "snake_case" names or
|
||||
enabling `omitempty` behavior by default.
|
||||
|
||||
## Usage
|
||||
```sh
|
||||
# install
|
||||
go get -u github.com/mailru/easyjson/...
|
||||
|
||||
# run
|
||||
easyjson -all <file>.go
|
||||
```
|
||||
|
||||
The above will generate `<file>_easyjson.go` containing the appropriate marshaler and
|
||||
unmarshaler funcs for all structs contained in `<file>.go`.
|
||||
|
||||
Please note that easyjson requires a full Go build environment and the `GOPATH`
|
||||
environment variable to be set. This is because easyjson code generation
|
||||
invokes `go run` on a temporary file (an approach to code generation borrowed
|
||||
from [ffjson](https://github.com/pquerna/ffjson)).
|
||||
|
||||
## Options
|
||||
```txt
|
||||
Usage of easyjson:
|
||||
-all
|
||||
generate marshaler/unmarshalers for all structs in a file
|
||||
-build_tags string
|
||||
build tags to add to generated file
|
||||
-leave_temps
|
||||
do not delete temporary files
|
||||
-no_std_marshalers
|
||||
don't generate MarshalJSON/UnmarshalJSON funcs
|
||||
-noformat
|
||||
do not run 'gofmt -w' on output file
|
||||
-omit_empty
|
||||
omit empty fields by default
|
||||
-output_filename string
|
||||
specify the filename of the output
|
||||
-pkg
|
||||
process the whole package instead of just the given file
|
||||
-snake_case
|
||||
use snake_case names instead of CamelCase by default
|
||||
-lower_camel_case
|
||||
use lowerCamelCase instead of CamelCase by default
|
||||
-stubs
|
||||
only generate stubs for marshaler/unmarshaler funcs
|
||||
-disallow_unknown_fields
|
||||
return error if some unknown field in json appeared
|
||||
```
|
||||
|
||||
Using `-all` will generate marshalers/unmarshalers for all Go structs in the
|
||||
file. If `-all` is not provided, then only those structs whose preceding
|
||||
comment starts with `easyjson:json` will have marshalers/unmarshalers
|
||||
generated. For example:
|
||||
|
||||
```go
|
||||
//easyjson:json
|
||||
type A struct {}
|
||||
```
|
||||
|
||||
Additional option notes:
|
||||
|
||||
* `-snake_case` tells easyjson to generate snake\_case field names by default
|
||||
(unless overridden by a field tag). The CamelCase to snake\_case conversion
|
||||
algorithm should work in most cases (ie, HTTPVersion will be converted to
|
||||
"http_version").
|
||||
|
||||
* `-build_tags` will add the specified build tags to generated Go sources.
|
||||
|
||||
## Generated Marshaler/Unmarshaler Funcs
|
||||
|
||||
For Go struct types, easyjson generates the funcs `MarshalEasyJSON` /
|
||||
`UnmarshalEasyJSON` for marshaling/unmarshaling JSON. In turn, these satisify
|
||||
the `easyjson.Marshaler` and `easyjson.Unmarshaler` interfaces and when used in
|
||||
conjunction with `easyjson.Marshal` / `easyjson.Unmarshal` avoid unnecessary
|
||||
reflection / type assertions during marshaling/unmarshaling to/from JSON for Go
|
||||
structs.
|
||||
|
||||
easyjson also generates `MarshalJSON` and `UnmarshalJSON` funcs for Go struct
|
||||
types compatible with the standard `json.Marshaler` and `json.Unmarshaler`
|
||||
interfaces. Please be aware that using the standard `json.Marshal` /
|
||||
`json.Unmarshal` for marshaling/unmarshaling will incur a significant
|
||||
performance penalty when compared to using `easyjson.Marshal` /
|
||||
`easyjson.Unmarshal`.
|
||||
|
||||
Additionally, easyjson exposes utility funcs that use the `MarshalEasyJSON` and
|
||||
`UnmarshalEasyJSON` for marshaling/unmarshaling to and from standard readers
|
||||
and writers. For example, easyjson provides `easyjson.MarshalToHTTPResponseWriter`
|
||||
which marshals to the standard `http.ResponseWriter`. Please see the [GoDoc
|
||||
listing](https://godoc.org/github.com/mailru/easyjson) for the full listing of
|
||||
utility funcs that are available.
|
||||
|
||||
## Controlling easyjson Marshaling and Unmarshaling Behavior
|
||||
|
||||
Go types can provide their own `MarshalEasyJSON` and `UnmarshalEasyJSON` funcs
|
||||
that satisify the `easyjson.Marshaler` / `easyjson.Unmarshaler` interfaces.
|
||||
These will be used by `easyjson.Marshal` and `easyjson.Unmarshal` when defined
|
||||
for a Go type.
|
||||
|
||||
Go types can also satisify the `easyjson.Optional` interface, which allows the
|
||||
type to define its own `omitempty` logic.
|
||||
|
||||
## Type Wrappers
|
||||
|
||||
easyjson provides additional type wrappers defined in the `easyjson/opt`
|
||||
package. These wrap the standard Go primitives and in turn satisify the
|
||||
easyjson interfaces.
|
||||
|
||||
The `easyjson/opt` type wrappers are useful when needing to distinguish between
|
||||
a missing value and/or when needing to specifying a default value. Type
|
||||
wrappers allow easyjson to avoid additional pointers and heap allocations and
|
||||
can significantly increase performance when used properly.
|
||||
|
||||
## Memory Pooling
|
||||
|
||||
easyjson uses a buffer pool that allocates data in increasing chunks from 128
|
||||
to 32768 bytes. Chunks of 512 bytes and larger will be reused with the help of
|
||||
`sync.Pool`. The maximum size of a chunk is bounded to reduce redundant memory
|
||||
allocation and to allow larger reusable buffers.
|
||||
|
||||
easyjson's custom allocation buffer pool is defined in the `easyjson/buffer`
|
||||
package, and the default behavior pool behavior can be modified (if necessary)
|
||||
through a call to `buffer.Init()` prior to any marshaling or unmarshaling.
|
||||
Please see the [GoDoc listing](https://godoc.org/github.com/mailru/easyjson/buffer)
|
||||
for more information.
|
||||
|
||||
## Issues, Notes, and Limitations
|
||||
|
||||
* easyjson is still early in its development. As such, there are likely to be
|
||||
bugs and missing features when compared to `encoding/json`. In the case of a
|
||||
missing feature or bug, please create a GitHub issue. Pull requests are
|
||||
welcome!
|
||||
|
||||
* Unlike `encoding/json`, object keys are case-sensitive. Case-insensitive
|
||||
matching is not currently provided due to the significant performance hit
|
||||
when doing case-insensitive key matching. In the future, case-insensitive
|
||||
object key matching may be provided via an option to the generator.
|
||||
|
||||
* easyjson makes use of `unsafe`, which simplifies the code and
|
||||
provides significant performance benefits by allowing no-copy
|
||||
conversion from `[]byte` to `string`. That said, `unsafe` is used
|
||||
only when unmarshaling and parsing JSON, and any `unsafe` operations
|
||||
/ memory allocations done will be safely deallocated by
|
||||
easyjson. Set the build tag `easyjson_nounsafe` to compile it
|
||||
without `unsafe`.
|
||||
|
||||
* easyjson is compatible with Google App Engine. The `appengine` build
|
||||
tag (set by App Engine's environment) will automatically disable the
|
||||
use of `unsafe`, which is not allowed in App Engine's Standard
|
||||
Environment. Note that the use with App Engine is still experimental.
|
||||
|
||||
* Floats are formatted using the default precision from Go's `strconv` package.
|
||||
As such, easyjson will not correctly handle high precision floats when
|
||||
marshaling/unmarshaling JSON. Note, however, that there are very few/limited
|
||||
uses where this behavior is not sufficient for general use. That said, a
|
||||
different package may be needed if precise marshaling/unmarshaling of high
|
||||
precision floats to/from JSON is required.
|
||||
|
||||
* While unmarshaling, the JSON parser does the minimal amount of work needed to
|
||||
skip over unmatching parens, and as such full validation is not done for the
|
||||
entire JSON value being unmarshaled/parsed.
|
||||
|
||||
* Currently there is no true streaming support for encoding/decoding as
|
||||
typically for many uses/protocols the final, marshaled length of the JSON
|
||||
needs to be known prior to sending the data. Currently this is not possible
|
||||
with easyjson's architecture.
|
||||
|
||||
* easyjson parser and codegen based on reflection, so it wont works on `package main`
|
||||
files, because they cant be imported by parser.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Most benchmarks were done using the example
|
||||
[13kB example JSON](https://dev.twitter.com/rest/reference/get/search/tweets)
|
||||
(9k after eliminating whitespace). This example is similar to real-world data,
|
||||
is well-structured, and contains a healthy variety of different types, making
|
||||
it ideal for JSON serialization benchmarks.
|
||||
|
||||
Note:
|
||||
|
||||
* For small request benchmarks, an 80 byte portion of the above example was
|
||||
used.
|
||||
|
||||
* For large request marshaling benchmarks, a struct containing 50 regular
|
||||
samples was used, making a ~500kB output JSON.
|
||||
|
||||
* Benchmarks are showing the results of easyjson's default behaviour,
|
||||
which makes use of `unsafe`.
|
||||
|
||||
Benchmarks are available in the repository and can be run by invoking `make`.
|
||||
|
||||
### easyjson vs. encoding/json
|
||||
|
||||
easyjson is roughly 5-6 times faster than the standard `encoding/json` for
|
||||
unmarshaling, and 3-4 times faster for non-concurrent marshaling. Concurrent
|
||||
marshaling is 6-7x faster if marshaling to a writer.
|
||||
|
||||
### easyjson vs. ffjson
|
||||
|
||||
easyjson uses the same approach for JSON marshaling as
|
||||
[ffjson](https://github.com/pquerna/ffjson), but takes a significantly
|
||||
different approach to lexing and parsing JSON during unmarshaling. This means
|
||||
easyjson is roughly 2-3x faster for unmarshaling and 1.5-2x faster for
|
||||
non-concurrent unmarshaling.
|
||||
|
||||
As of this writing, `ffjson` seems to have issues when used concurrently:
|
||||
specifically, large request pooling hurts `ffjson`'s performance and causes
|
||||
scalability issues. These issues with `ffjson` can likely be fixed, but as of
|
||||
writing remain outstanding/known issues with `ffjson`.
|
||||
|
||||
easyjson and `ffjson` have similar performance for small requests, however
|
||||
easyjson outperforms `ffjson` by roughly 2-5x times for large requests when
|
||||
used with a writer.
|
||||
|
||||
### easyjson vs. go/codec
|
||||
|
||||
[go/codec](https://github.com/ugorji/go) provides
|
||||
compile-time helpers for JSON generation. In this case, helpers do not work
|
||||
like marshalers as they are encoding-independent.
|
||||
|
||||
easyjson is generally 2x faster than `go/codec` for non-concurrent benchmarks
|
||||
and about 3x faster for concurrent encoding (without marshaling to a writer).
|
||||
|
||||
In an attempt to measure marshaling performance of `go/codec` (as opposed to
|
||||
allocations/memcpy/writer interface invocations), a benchmark was done with
|
||||
resetting length of a byte slice rather than resetting the whole slice to nil.
|
||||
However, the optimization in this exact form may not be applicable in practice,
|
||||
since the memory is not freed between marshaling operations.
|
||||
|
||||
### easyjson vs 'ujson' python module
|
||||
|
||||
[ujson](https://github.com/esnme/ultrajson) is using C code for parsing, so it
|
||||
is interesting to see how plain golang compares to that. It is imporant to note
|
||||
that the resulting object for python is slower to access, since the library
|
||||
parses JSON object into dictionaries.
|
||||
|
||||
easyjson is slightly faster for unmarshaling and 2-3x faster than `ujson` for
|
||||
marshaling.
|
||||
|
||||
### Benchmark Results
|
||||
|
||||
`ffjson` results are from February 4th, 2016, using the latest `ffjson` and go1.6.
|
||||
`go/codec` results are from March 4th, 2016, using the latest `go/codec` and go1.6.
|
||||
|
||||
#### Unmarshaling
|
||||
|
||||
| lib | json size | MB/s | allocs/op | B/op |
|
||||
|:---------|:----------|-----:|----------:|------:|
|
||||
| standard | regular | 22 | 218 | 10229 |
|
||||
| standard | small | 9.7 | 14 | 720 |
|
||||
| | | | | |
|
||||
| easyjson | regular | 125 | 128 | 9794 |
|
||||
| easyjson | small | 67 | 3 | 128 |
|
||||
| | | | | |
|
||||
| ffjson | regular | 66 | 141 | 9985 |
|
||||
| ffjson | small | 17.6 | 10 | 488 |
|
||||
| | | | | |
|
||||
| codec | regular | 55 | 434 | 19299 |
|
||||
| codec | small | 29 | 7 | 336 |
|
||||
| | | | | |
|
||||
| ujson | regular | 103 | N/A | N/A |
|
||||
|
||||
#### Marshaling, one goroutine.
|
||||
|
||||
| lib | json size | MB/s | allocs/op | B/op |
|
||||
|:----------|:----------|-----:|----------:|------:|
|
||||
| standard | regular | 75 | 9 | 23256 |
|
||||
| standard | small | 32 | 3 | 328 |
|
||||
| standard | large | 80 | 17 | 1.2M |
|
||||
| | | | | |
|
||||
| easyjson | regular | 213 | 9 | 10260 |
|
||||
| easyjson* | regular | 263 | 8 | 742 |
|
||||
| easyjson | small | 125 | 1 | 128 |
|
||||
| easyjson | large | 212 | 33 | 490k |
|
||||
| easyjson* | large | 262 | 25 | 2879 |
|
||||
| | | | | |
|
||||
| ffjson | regular | 122 | 153 | 21340 |
|
||||
| ffjson** | regular | 146 | 152 | 4897 |
|
||||
| ffjson | small | 36 | 5 | 384 |
|
||||
| ffjson** | small | 64 | 4 | 128 |
|
||||
| ffjson | large | 134 | 7317 | 818k |
|
||||
| ffjson** | large | 125 | 7320 | 827k |
|
||||
| | | | | |
|
||||
| codec | regular | 80 | 17 | 33601 |
|
||||
| codec*** | regular | 108 | 9 | 1153 |
|
||||
| codec | small | 42 | 3 | 304 |
|
||||
| codec*** | small | 56 | 1 | 48 |
|
||||
| codec | large | 73 | 483 | 2.5M |
|
||||
| codec*** | large | 103 | 451 | 66007 |
|
||||
| | | | | |
|
||||
| ujson | regular | 92 | N/A | N/A |
|
||||
|
||||
\* marshaling to a writer,
|
||||
\*\* using `ffjson.Pool()`,
|
||||
\*\*\* reusing output slice instead of resetting it to nil
|
||||
|
||||
#### Marshaling, concurrent.
|
||||
|
||||
| lib | json size | MB/s | allocs/op | B/op |
|
||||
|:----------|:----------|-----:|----------:|------:|
|
||||
| standard | regular | 252 | 9 | 23257 |
|
||||
| standard | small | 124 | 3 | 328 |
|
||||
| standard | large | 289 | 17 | 1.2M |
|
||||
| | | | | |
|
||||
| easyjson | regular | 792 | 9 | 10597 |
|
||||
| easyjson* | regular | 1748 | 8 | 779 |
|
||||
| easyjson | small | 333 | 1 | 128 |
|
||||
| easyjson | large | 718 | 36 | 548k |
|
||||
| easyjson* | large | 2134 | 25 | 4957 |
|
||||
| | | | | |
|
||||
| ffjson | regular | 301 | 153 | 21629 |
|
||||
| ffjson** | regular | 707 | 152 | 5148 |
|
||||
| ffjson | small | 62 | 5 | 384 |
|
||||
| ffjson** | small | 282 | 4 | 128 |
|
||||
| ffjson | large | 438 | 7330 | 1.0M |
|
||||
| ffjson** | large | 131 | 7319 | 820k |
|
||||
| | | | | |
|
||||
| codec | regular | 183 | 17 | 33603 |
|
||||
| codec*** | regular | 671 | 9 | 1157 |
|
||||
| codec | small | 147 | 3 | 304 |
|
||||
| codec*** | small | 299 | 1 | 48 |
|
||||
| codec | large | 190 | 483 | 2.5M |
|
||||
| codec*** | large | 752 | 451 | 77574 |
|
||||
|
||||
\* marshaling to a writer,
|
||||
\*\* using `ffjson.Pool()`,
|
||||
\*\*\* reusing output slice instead of resetting it to nil
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
module github.com/mailru/easyjson
|
||||
|
||||
go 1.12
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
// Package easyjson contains marshaler/unmarshaler interfaces and helper functions.
|
||||
package easyjson
|
||||
|
||||
import (
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
"strconv"
|
||||
|
||||
"github.com/mailru/easyjson/jlexer"
|
||||
"github.com/mailru/easyjson/jwriter"
|
||||
)
|
||||
|
||||
// Marshaler is an easyjson-compatible marshaler interface.
|
||||
type Marshaler interface {
|
||||
MarshalEasyJSON(w *jwriter.Writer)
|
||||
}
|
||||
|
||||
// Marshaler is an easyjson-compatible unmarshaler interface.
|
||||
type Unmarshaler interface {
|
||||
UnmarshalEasyJSON(w *jlexer.Lexer)
|
||||
}
|
||||
|
||||
// Optional defines an undefined-test method for a type to integrate with 'omitempty' logic.
|
||||
type Optional interface {
|
||||
IsDefined() bool
|
||||
}
|
||||
|
||||
// Marshal returns data as a single byte slice. Method is suboptimal as the data is likely to be copied
|
||||
// from a chain of smaller chunks.
|
||||
func Marshal(v Marshaler) ([]byte, error) {
|
||||
w := jwriter.Writer{}
|
||||
v.MarshalEasyJSON(&w)
|
||||
return w.BuildBytes()
|
||||
}
|
||||
|
||||
// MarshalToWriter marshals the data to an io.Writer.
|
||||
func MarshalToWriter(v Marshaler, w io.Writer) (written int, err error) {
|
||||
jw := jwriter.Writer{}
|
||||
v.MarshalEasyJSON(&jw)
|
||||
return jw.DumpTo(w)
|
||||
}
|
||||
|
||||
// MarshalToHTTPResponseWriter sets Content-Length and Content-Type headers for the
|
||||
// http.ResponseWriter, and send the data to the writer. started will be equal to
|
||||
// false if an error occurred before any http.ResponseWriter methods were actually
|
||||
// invoked (in this case a 500 reply is possible).
|
||||
func MarshalToHTTPResponseWriter(v Marshaler, w http.ResponseWriter) (started bool, written int, err error) {
|
||||
jw := jwriter.Writer{}
|
||||
v.MarshalEasyJSON(&jw)
|
||||
if jw.Error != nil {
|
||||
return false, 0, jw.Error
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("Content-Length", strconv.Itoa(jw.Size()))
|
||||
|
||||
started = true
|
||||
written, err = jw.DumpTo(w)
|
||||
return
|
||||
}
|
||||
|
||||
// Unmarshal decodes the JSON in data into the object.
|
||||
func Unmarshal(data []byte, v Unmarshaler) error {
|
||||
l := jlexer.Lexer{Data: data}
|
||||
v.UnmarshalEasyJSON(&l)
|
||||
return l.Error()
|
||||
}
|
||||
|
||||
// UnmarshalFromReader reads all the data in the reader and decodes as JSON into the object.
|
||||
func UnmarshalFromReader(r io.Reader, v Unmarshaler) error {
|
||||
data, err := ioutil.ReadAll(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
l := jlexer.Lexer{Data: data}
|
||||
v.UnmarshalEasyJSON(&l)
|
||||
return l.Error()
|
||||
}
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
package easyjson
|
||||
|
||||
import (
|
||||
"github.com/mailru/easyjson/jlexer"
|
||||
"github.com/mailru/easyjson/jwriter"
|
||||
)
|
||||
|
||||
// RawMessage is a raw piece of JSON (number, string, bool, object, array or
|
||||
// null) that is extracted without parsing and output as is during marshaling.
|
||||
type RawMessage []byte
|
||||
|
||||
// MarshalEasyJSON does JSON marshaling using easyjson interface.
|
||||
func (v *RawMessage) MarshalEasyJSON(w *jwriter.Writer) {
|
||||
if len(*v) == 0 {
|
||||
w.RawString("null")
|
||||
} else {
|
||||
w.Raw(*v, nil)
|
||||
}
|
||||
}
|
||||
|
||||
// UnmarshalEasyJSON does JSON unmarshaling using easyjson interface.
|
||||
func (v *RawMessage) UnmarshalEasyJSON(l *jlexer.Lexer) {
|
||||
*v = RawMessage(l.Raw())
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements encoding/json.Unmarshaler interface.
|
||||
func (v *RawMessage) UnmarshalJSON(data []byte) error {
|
||||
*v = data
|
||||
return nil
|
||||
}
|
||||
|
||||
var nullBytes = []byte("null")
|
||||
|
||||
// MarshalJSON implements encoding/json.Marshaler interface.
|
||||
func (v RawMessage) MarshalJSON() ([]byte, error) {
|
||||
if len(v) == 0 {
|
||||
return nullBytes, nil
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// IsDefined is required for integration with omitempty easyjson logic.
|
||||
func (v *RawMessage) IsDefined() bool {
|
||||
return len(*v) > 0
|
||||
}
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# Generated by FOSSA CLI (https://github.com/fossas/fossa-cli)
|
||||
# Visit https://fossa.io to learn more
|
||||
|
||||
version: 1
|
||||
cli:
|
||||
server: https://app.fossa.io
|
||||
fetcher: git
|
||||
project: git@github.com:olivere/elastic.git
|
||||
analyze:
|
||||
modules:
|
||||
- name: github.com/olivere/elastic
|
||||
path: .
|
||||
target: github.com/olivere/elastic
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/config
|
||||
path: ./config
|
||||
target: github.com/olivere/elastic/config
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/uritemplates
|
||||
path: ./uritemplates
|
||||
target: github.com/olivere/elastic/uritemplates
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/trace/opencensus
|
||||
path: ./trace/opencensus
|
||||
target: github.com/olivere/elastic/trace/opencensus
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/trace/opentracing
|
||||
path: ./trace/opentracing
|
||||
target: github.com/olivere/elastic/trace/opentracing
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/aws
|
||||
path: ./aws
|
||||
target: github.com/olivere/elastic/aws
|
||||
type: go
|
||||
- name: github.com/olivere/elastic/aws/v4
|
||||
path: ./aws/v4
|
||||
target: github.com/olivere/elastic/aws/v4
|
||||
type: go
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
# Compiled Object files, Static and Dynamic libs (Shared Objects)
|
||||
*.o
|
||||
*.a
|
||||
*.so
|
||||
|
||||
# Folders
|
||||
_obj
|
||||
_test
|
||||
|
||||
# Architecture specific extensions/prefixes
|
||||
*.[568vq]
|
||||
[568vq].out
|
||||
|
||||
*.cgo1.go
|
||||
*.cgo2.c
|
||||
_cgo_defun.c
|
||||
_cgo_gotypes.go
|
||||
_cgo_export.*
|
||||
|
||||
_testmain.go
|
||||
|
||||
*.exe
|
||||
|
||||
/.vscode/
|
||||
/.idea/
|
||||
/debug.test
|
||||
/generator
|
||||
/cluster-test/cluster-test
|
||||
/cluster-test/*.log
|
||||
/cluster-test/es-chaos-monkey
|
||||
/go.sum
|
||||
/spec
|
||||
/tmp
|
||||
/CHANGELOG-3.0.html
|
||||
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
sudo: required
|
||||
language: go
|
||||
go:
|
||||
- "1.12.x"
|
||||
- "1.13.x"
|
||||
- tip
|
||||
matrix:
|
||||
allow_failures:
|
||||
- go: tip
|
||||
env:
|
||||
- GO111MODULE=on
|
||||
- GO111MODULE=off
|
||||
addons:
|
||||
apt:
|
||||
update: true
|
||||
packages:
|
||||
- docker-ce
|
||||
services:
|
||||
- docker
|
||||
before_install:
|
||||
- if [[ "$TRAVIS_OS_NAME" == "linux" && ! $(which nc) ]] ; then sudo apt-get install -y netcat ; fi
|
||||
- sudo sysctl -w vm.max_map_count=262144
|
||||
- docker-compose pull
|
||||
- docker-compose up -d
|
||||
- go get -u github.com/google/go-cmp/cmp
|
||||
- go get -u github.com/fortytw2/leaktest
|
||||
- go get . ./aws/... ./config/... ./trace/... ./uritemplates/...
|
||||
- while ! nc -z localhost 9200; do sleep 1; done
|
||||
- while ! nc -z localhost 9210; do sleep 1; done
|
||||
install: true # ignore the go get -t -v ./...
|
||||
script:
|
||||
- go test -race -deprecations -strict-decoder -v . ./aws/... ./config/... ./trace/... ./uritemplates/...
|
||||
|
|
@ -0,0 +1,363 @@
|
|||
# Elastic 3.0
|
||||
|
||||
Elasticsearch 2.0 comes with some [breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking-changes-2.0.html). You will probably need to upgrade your application and/or rewrite part of it due to those changes.
|
||||
|
||||
We use that window of opportunity to also update Elastic (the Go client) from version 2.0 to 3.0. This will introduce both changes due to the Elasticsearch 2.0 update as well as changes that make Elastic cleaner by removing some old cruft.
|
||||
|
||||
So, to summarize:
|
||||
|
||||
1. Elastic 2.0 is compatible with Elasticsearch 1.7+ and is still actively maintained.
|
||||
2. Elastic 3.0 is compatible with Elasticsearch 2.0+ and will soon become the new master branch.
|
||||
|
||||
The rest of the document is a list of all changes in Elastic 3.0.
|
||||
|
||||
## Pointer types
|
||||
|
||||
All types have changed to be pointer types, not value types. This not only is cleaner but also simplifies the API as illustrated by the following example:
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
q := elastic.NewMatchAllQuery()
|
||||
res, err := elastic.Search("one").Query(&q).Do() // notice the & here
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
q := elastic.NewMatchAllQuery()
|
||||
res, err := elastic.Search("one").Query(q).Do() // no more &
|
||||
// ... which can be simplified as:
|
||||
res, err := elastic.Search("one").Query(elastic.NewMatchAllQuery()).Do()
|
||||
```
|
||||
|
||||
It also helps to prevent [subtle issues](https://github.com/olivere/elastic/issues/115#issuecomment-130753046).
|
||||
|
||||
## Query/filter merge
|
||||
|
||||
One of the biggest changes in Elasticsearch 2.0 is the [merge of queries and filters](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_queries_and_filters_merged). In Elasticsearch 1.x, you had a whole range of queries and filters that were basically identical (e.g. `term_query` and `term_filter`).
|
||||
|
||||
The practical aspect of the merge is that you can now basically use queries where once you had to use filters instead. For Elastic 3.0 this means: We could remove a whole bunch of files. Yay!
|
||||
|
||||
Notice that some methods still come by "filter", e.g. `PostFilter`. However, they accept a `Query` now when they used to accept a `Filter` before.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
q := elastic.NewMatchAllQuery()
|
||||
f := elastic.NewTermFilter("tag", "important")
|
||||
res, err := elastic.Search().Index("one").Query(&q).PostFilter(f)
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
q := elastic.NewMatchAllQuery()
|
||||
f := elastic.NewTermQuery("tag", "important") // it's a query now!
|
||||
res, err := elastic.Search().Index("one").Query(q).PostFilter(f)
|
||||
```
|
||||
|
||||
## Facets are removed
|
||||
|
||||
[Facets have been removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_removed_features.html#_facets_have_been_removed) in Elasticsearch 2.0. You need to use aggregations now.
|
||||
|
||||
## Errors
|
||||
|
||||
Elasticsearch 2.0 returns more information about an error in the HTTP response body. Elastic 3.0 now reads this information and makes it accessible by the consumer.
|
||||
|
||||
Errors and all its details are now returned in [`Error`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59).
|
||||
|
||||
### HTTP Status 404 (Not Found)
|
||||
|
||||
When Elasticsearch does not find an entity or an index, it generally returns HTTP status code 404. In Elastic 2.0 this was a valid result and didn't raise an error from the `Do` functions. This has now changed in Elastic 3.0.
|
||||
|
||||
Starting with Elastic 3.0, there are only two types of responses considered successful. First, responses with HTTP status codes [200..299]. Second, HEAD requests which return HTTP status 404. The latter is used by Elasticsearch to e.g. check for existence of indices or documents. All other responses will return an error.
|
||||
|
||||
To check for HTTP Status 404 (with non-HEAD requests), e.g. when trying to get or delete a missing document, you can use the [`IsNotFound`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L84) helper (see below).
|
||||
|
||||
The following example illustrates how to check for a missing document in Elastic 2.0 and what has changed in 3.0.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
|
||||
if err != nil {
|
||||
// Something else went wrong (but 404 is NOT an error in Elastic 2.0)
|
||||
}
|
||||
if !res.Found {
|
||||
// Document has not been found
|
||||
}
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
|
||||
if err != nil {
|
||||
if elastic.IsNotFound(err) {
|
||||
// Document has not been found
|
||||
} else {
|
||||
// Something else went wrong
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### HTTP Status 408 (Timeouts)
|
||||
|
||||
Elasticsearch now responds with HTTP status code 408 (Timeout) when a request fails due to a timeout. E.g. if you specify a timeout with the Cluster Health API, the HTTP response status will be 408 if the timeout is raised. See [here](https://github.com/elastic/elasticsearch/commit/fe3179d9cccb569784434b2135ca9ae13d5158d3) for the specific commit to the Cluster Health API.
|
||||
|
||||
To check for HTTP Status 408, we introduced the [`IsTimeout`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L101) helper.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
|
||||
if err != nil {
|
||||
// ...
|
||||
}
|
||||
if health.TimedOut {
|
||||
// We have a timeout
|
||||
}
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
|
||||
if elastic.IsTimeout(err) {
|
||||
// We have a timeout
|
||||
}
|
||||
```
|
||||
|
||||
### Bulk Errors
|
||||
|
||||
The error response of a bulk operation used to be a simple string in Elasticsearch 1.x.
|
||||
In Elasticsearch 2.0, it returns a structured JSON object with a lot more details about the error.
|
||||
These errors are now captured in an object of type [`ErrorDetails`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59) which is used in [`BulkResponseItem`](https://github.com/olivere/elastic/blob/release-branch.v3/bulk.go#L206).
|
||||
|
||||
### Removed specific Elastic errors
|
||||
|
||||
The specific error types `ErrMissingIndex`, `ErrMissingType`, and `ErrMissingId` have been removed. They were only used by `DeleteService` and are replaced by a generic error message.
|
||||
|
||||
## Numeric types
|
||||
|
||||
Elastic 3.0 has settled to use `float64` everywhere. It used to be a mix of `float32` and `float64` in Elastic 2.0. E.g. all boostable queries in Elastic 3.0 now have a boost type of `float64` where it used to be `float32`.
|
||||
|
||||
## Pluralization
|
||||
|
||||
Some services accept zero, one or more indices or types to operate on.
|
||||
E.g. in the `SearchService` accepts a list of zero, one, or more indices to
|
||||
search and therefor had a func called `Index(index string)` and a func
|
||||
called `Indices(indices ...string)`.
|
||||
|
||||
Elastic 3.0 now only uses the singular form that, when applicable, accepts a
|
||||
variadic type. E.g. in the case of the `SearchService`, you now only have
|
||||
one func with the following signature: `Index(indices ...string)`.
|
||||
|
||||
Notice this is only limited to `Index(...)` and `Type(...)`. There are other
|
||||
services with variadic functions. These have not been changed.
|
||||
|
||||
## Multiple calls to variadic functions
|
||||
|
||||
Some services with variadic functions have cleared the underlying slice when
|
||||
called while other services just add to the existing slice. This has now been
|
||||
normalized to always add to the underlying slice.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
// Would only cleared scroll id "two"
|
||||
// because ScrollId cleared the values when called multiple times
|
||||
client.ClearScroll().ScrollId("one").ScrollId("two").Do()
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
// Now (correctly) clears both scroll id "one" and "two"
|
||||
// because ScrollId no longer clears the values when called multiple times
|
||||
client.ClearScroll().ScrollId("one").ScrollId("two").Do()
|
||||
```
|
||||
|
||||
## Ping service requires URL
|
||||
|
||||
The `Ping` service raised some issues because it is different from all
|
||||
other services. If not explicitly given a URL, it always pings `127.0.0.1:9200`.
|
||||
|
||||
Users expected to ping the cluster, but that is not possible as the cluster
|
||||
can be a set of many nodes: So which node do we ping then?
|
||||
|
||||
To make it more clear, the `Ping` function on the client now requires users
|
||||
to explicitly set the URL of the node to ping.
|
||||
|
||||
## Meta fields
|
||||
|
||||
Many of the meta fields e.g. `_parent` or `_routing` are now
|
||||
[part of the top-level of a document](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_mapping_changes.html#migration-meta-fields)
|
||||
and are no longer returned as parts of the `fields` object. We had to change
|
||||
larger parts of e.g. the `Reindexer` to get it to work seamlessly with Elasticsearch 2.0.
|
||||
|
||||
Notice that all stored meta-fields are now [returned by default](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_crud_and_routing_changes.html#_all_stored_meta_fields_returned_by_default).
|
||||
|
||||
## HasParentQuery / HasChildQuery
|
||||
|
||||
`NewHasParentQuery` and `NewHasChildQuery` must now include both parent/child type and query. It is now in line with the Java API.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
allQ := elastic.NewMatchAllQuery()
|
||||
q := elastic.NewHasChildFilter("tweet").Query(&allQ)
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
q := elastic.NewHasChildQuery("tweet", elastic.NewMatchAllQuery())
|
||||
```
|
||||
|
||||
## SetBasicAuth client option
|
||||
|
||||
You can now tell Elastic to pass HTTP Basic Auth credentials with each request. In previous versions of Elastic you had to set up your own `http.Transport` to do this. This should make it more convenient to use Elastic in combination with [Shield](https://www.elastic.co/products/shield) in its [basic setup](https://www.elastic.co/guide/en/shield/current/enable-basic-auth.html).
|
||||
|
||||
Example:
|
||||
|
||||
```go
|
||||
client, err := elastic.NewClient(elastic.SetBasicAuth("user", "secret"))
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
## Delete-by-Query API
|
||||
|
||||
The Delete-by-Query API is [a plugin now](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_removed_features.html#_delete_by_query_is_now_a_plugin). It is no longer core part of Elasticsearch. You can [install it as a plugin as described here](https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugins-delete-by-query.html).
|
||||
|
||||
Elastic 3.0 still contains the `DeleteByQueryService`, but you need to install the plugin first. If you don't install it and use `DeleteByQueryService` you will most probably get a 404.
|
||||
|
||||
An older version of this document stated the following:
|
||||
|
||||
> Elastic 3.0 still contains the `DeleteByQueryService` but it will fail with `ErrPluginNotFound` when the plugin is not installed.
|
||||
>
|
||||
> Example for Elastic 3.0 (new):
|
||||
>
|
||||
> ```go
|
||||
> _, err := client.DeleteByQuery().Query(elastic.NewTermQuery("client", "1")).Do()
|
||||
> if err == elastic.ErrPluginNotFound {
|
||||
> // Delete By Query API is not available
|
||||
> }
|
||||
> ```
|
||||
|
||||
I have decided that this is not a good way to handle the case of a missing plugin. The main reason is that with this logic, you'd always have to check if the plugin is missing in case of an error. This is not only slow, but it also puts logic into a service where it should really be just opaque and return the response of Elasticsearch.
|
||||
|
||||
If you rely on certain plugins to be installed, you should check on startup. That's where the following two helpers come into play.
|
||||
|
||||
## HasPlugin and SetRequiredPlugins
|
||||
|
||||
Some of the core functionality of Elasticsearch has now been moved into plugins. E.g. the Delete-by-Query API is [a plugin now](https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugins-delete-by-query.html).
|
||||
|
||||
You need to make sure to add these plugins to your Elasticsearch installation to still be able to use the `DeleteByQueryService`. You can test this now with the `HasPlugin(name string)` helper in the client.
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
err, found := client.HasPlugin("delete-by-query")
|
||||
if err == nil && found {
|
||||
// ... Delete By Query API is available
|
||||
}
|
||||
```
|
||||
|
||||
To simplify this process, there is now a `SetRequiredPlugins` helper that can be passed as an option func when creating a new client. If the plugin is not installed, the client wouldn't be created in the first place.
|
||||
|
||||
```go
|
||||
// Will raise an error if the "delete-by-query" plugin is NOT installed
|
||||
client, err := elastic.NewClient(elastic.SetRequiredPlugins("delete-by-query"))
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
Notice that there also is a way to define [mandatory plugins](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#_mandatory_plugins) in the Elasticsearch configuration file.
|
||||
|
||||
## Common Query has been renamed to Common Terms Query
|
||||
|
||||
The `CommonQuery` has been renamed to `CommonTermsQuery` to be in line with the [Java API](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_java_api_changes.html#_query_filter_refactoring).
|
||||
|
||||
## Remove `MoreLikeThis` and `MoreLikeThisField`
|
||||
|
||||
The More Like This API and the More Like This Field query [have been removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_more_like_this) and replaced with the `MoreLikeThisQuery`.
|
||||
|
||||
## Remove Filtered Query
|
||||
|
||||
With the merge of queries and filters, the [filtered query became deprecated](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_filtered_literal_query_and_literal_query_literal_filter_deprecated). While it is only deprecated and therefore still available in Elasticsearch 2.0, we have decided to remove it from Elastic 3.0. Why? Because we think that when you're already forced to rewrite many of your application code, it might be a good chance to get rid of things that are deprecated as well. So you might simply change your filtered query with a boolean query as [described here](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_filtered_literal_query_and_literal_query_literal_filter_deprecated).
|
||||
|
||||
## Remove FuzzyLikeThis and FuzzyLikeThisField
|
||||
|
||||
Both have been removed from Elasticsearch 2.0 as well.
|
||||
|
||||
## Remove LimitFilter
|
||||
|
||||
The `limit` filter is [deprecated in Elasticsearch 2.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_literal_limit_literal_filter_deprecated) and becomes a no-op. Now is a good chance to remove it from your application as well. Use the `terminate_after` parameter in your search [as described here](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/search-request-body.html) to achieve similar effects.
|
||||
|
||||
## Remove `_cache` and `_cache_key` from filters
|
||||
|
||||
Both have been [removed from Elasticsearch 2.0 as well](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_filter_auto_caching).
|
||||
|
||||
## Partial fields are gone
|
||||
|
||||
Partial fields are [removed in Elasticsearch 2.0](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_search_changes.html#_partial_fields) in favor of [source filtering](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/search-request-source-filtering.html).
|
||||
|
||||
## Scripting
|
||||
|
||||
A [`Script`](https://github.com/olivere/elastic/blob/release-branch.v3/script.go) type has been added to Elastic 3.0. In Elastic 2.0, there were various places (e.g. aggregations) where you could just add the script as a string, specify the scripting language, add parameters etc. With Elastic 3.0, you should now always use the `Script` type.
|
||||
|
||||
Example for Elastic 2.0 (old):
|
||||
|
||||
```go
|
||||
update, err := client.Update().Index("twitter").Type("tweet").Id("1").
|
||||
Script("ctx._source.retweets += num").
|
||||
ScriptParams(map[string]interface{}{"num": 1}).
|
||||
Upsert(map[string]interface{}{"retweets": 0}).
|
||||
Do()
|
||||
```
|
||||
|
||||
Example for Elastic 3.0 (new):
|
||||
|
||||
```go
|
||||
update, err := client.Update().Index("twitter").Type("tweet").Id("1").
|
||||
Script(elastic.NewScript("ctx._source.retweets += num").Param("num", 1)).
|
||||
Upsert(map[string]interface{}{"retweets": 0}).
|
||||
Do()
|
||||
```
|
||||
|
||||
## Cluster State
|
||||
|
||||
The combination of `Metric(string)` and `Metrics(...string)` has been replaced by a single func with the signature `Metric(...string)`.
|
||||
|
||||
## Unexported structs in response
|
||||
|
||||
Services generally return a typed response from a `Do` func. Those structs are exported so that they can be passed around in your own application. In Elastic 3.0 however, we changed that (most) sub-structs are now unexported, meaning: You can only pass around the whole response, not sub-structures of it. This makes it easier for restructuring responses according to the Elasticsearch API. See [`ClusterStateResponse`](https://github.com/olivere/elastic/blob/release-branch.v3/cluster_state.go#L182) as an example.
|
||||