Issue search support elasticsearch (#9428)

* Issue search support elasticsearch

* Fix lint

* Add indexer name on app.ini

* add a warnning on SearchIssuesByKeyword

* improve code
pull/10263/head
Lunny Xiao 3 years ago committed by GitHub
parent 17656021f1
commit 5dbf36f356
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 6
      .drone.yml
  2. 6
      custom/conf/app.ini.sample
  3. 6
      docs/content/doc/advanced/config-cheat-sheet.en-us.md
  4. 4
      docs/content/doc/advanced/config-cheat-sheet.zh-cn.md
  5. 1
      go.mod
  6. 10
      go.sum
  7. 2
      integrations/mysql.ini.tmpl
  8. 2
      modules/indexer/issues/bleve.go
  9. 230
      modules/indexer/issues/elastic_search.go
  10. 32
      modules/indexer/issues/indexer.go
  11. 30
      modules/setting/indexer.go
  12. 5
      vendor/github.com/mailru/easyjson/.gitignore
  13. 12
      vendor/github.com/mailru/easyjson/.travis.yml
  14. 52
      vendor/github.com/mailru/easyjson/Makefile
  15. 336
      vendor/github.com/mailru/easyjson/README.md
  16. 3
      vendor/github.com/mailru/easyjson/go.mod
  17. 78
      vendor/github.com/mailru/easyjson/helpers.go
  18. 45
      vendor/github.com/mailru/easyjson/raw.go
  19. 38
      vendor/github.com/olivere/elastic/v7/.fossa.yml
  20. 35
      vendor/github.com/olivere/elastic/v7/.gitignore
  21. 32
      vendor/github.com/olivere/elastic/v7/.travis.yml
  22. 363
      vendor/github.com/olivere/elastic/v7/CHANGELOG-3.0.md
  23. 195
      vendor/github.com/olivere/elastic/v7/CHANGELOG-5.0.md
  24. 18
      vendor/github.com/olivere/elastic/v7/CHANGELOG-6.0.md
  25. 55
      vendor/github.com/olivere/elastic/v7/CHANGELOG-7.0.md
  26. 46
      vendor/github.com/olivere/elastic/v7/CODE_OF_CONDUCT.md
  27. 40
      vendor/github.com/olivere/elastic/v7/CONTRIBUTING.md
  28. 193
      vendor/github.com/olivere/elastic/v7/CONTRIBUTORS
  29. 19
      vendor/github.com/olivere/elastic/v7/ISSUE_TEMPLATE.md
  30. 20
      vendor/github.com/olivere/elastic/v7/LICENSE
  31. 432
      vendor/github.com/olivere/elastic/v7/README.md
  32. 13
      vendor/github.com/olivere/elastic/v7/acknowledged_response.go
  33. 148
      vendor/github.com/olivere/elastic/v7/backoff.go
  34. 470
      vendor/github.com/olivere/elastic/v7/bulk.go
  35. 186
      vendor/github.com/olivere/elastic/v7/bulk_delete_request.go
  36. 266
      vendor/github.com/olivere/elastic/v7/bulk_delete_request_easyjson.go
  37. 260
      vendor/github.com/olivere/elastic/v7/bulk_index_request.go
  38. 306
      vendor/github.com/olivere/elastic/v7/bulk_index_request_easyjson.go
  39. 656
      vendor/github.com/olivere/elastic/v7/bulk_processor.go
  40. 17
      vendor/github.com/olivere/elastic/v7/bulk_request.go
  41. 334
      vendor/github.com/olivere/elastic/v7/bulk_update_request.go
  42. 493
      vendor/github.com/olivere/elastic/v7/bulk_update_request_easyjson.go
  43. 34
      vendor/github.com/olivere/elastic/v7/canonicalize.go
  44. 222
      vendor/github.com/olivere/elastic/v7/cat_aliases.go
  45. 239
      vendor/github.com/olivere/elastic/v7/cat_allocation.go
  46. 215
      vendor/github.com/olivere/elastic/v7/cat_count.go
  47. 211
      vendor/github.com/olivere/elastic/v7/cat_health.go
  48. 374
      vendor/github.com/olivere/elastic/v7/cat_indices.go
  49. 162
      vendor/github.com/olivere/elastic/v7/clear_scroll.go
  50. 2099
      vendor/github.com/olivere/elastic/v7/client.go
  51. 296
      vendor/github.com/olivere/elastic/v7/cluster_health.go
  52. 438
      vendor/github.com/olivere/elastic/v7/cluster_reroute.go
  53. 355
      vendor/github.com/olivere/elastic/v7/cluster_state.go
  54. 409
      vendor/github.com/olivere/elastic/v7/cluster_stats.go
  55. 87
      vendor/github.com/olivere/elastic/v7/config/config.go
  56. 9
      vendor/github.com/olivere/elastic/v7/config/doc.go
  57. 90
      vendor/github.com/olivere/elastic/v7/connection.go
  58. 381
      vendor/github.com/olivere/elastic/v7/count.go
  59. 38
      vendor/github.com/olivere/elastic/v7/decoder.go
  60. 305
      vendor/github.com/olivere/elastic/v7/delete.go
  61. 781
      vendor/github.com/olivere/elastic/v7/delete_by_query.go
  62. 51
      vendor/github.com/olivere/elastic/v7/doc.go
  63. 59
      vendor/github.com/olivere/elastic/v7/docker-compose.yml
  64. 42
      vendor/github.com/olivere/elastic/v7/docvalue_field.go
  65. 184
      vendor/github.com/olivere/elastic/v7/errors.go
  66. 236
      vendor/github.com/olivere/elastic/v7/exists.go
  67. 390
      vendor/github.com/olivere/elastic/v7/explain.go
  68. 90
      vendor/github.com/olivere/elastic/v7/fetch_source_context.go
  69. 257
      vendor/github.com/olivere/elastic/v7/field_caps.go
  70. 54
      vendor/github.com/olivere/elastic/v7/geo_point.go
  71. 317
      vendor/github.com/olivere/elastic/v7/get.go
  72. 18
      vendor/github.com/olivere/elastic/v7/go.mod
  73. 469
      vendor/github.com/olivere/elastic/v7/highlight.go
  74. 377
      vendor/github.com/olivere/elastic/v7/index.go
  75. 320
      vendor/github.com/olivere/elastic/v7/indices_analyze.go
  76. 214
      vendor/github.com/olivere/elastic/v7/indices_close.go
  77. 189
      vendor/github.com/olivere/elastic/v7/indices_create.go
  78. 184
      vendor/github.com/olivere/elastic/v7/indices_delete.go
  79. 180
      vendor/github.com/olivere/elastic/v7/indices_delete_template.go
  80. 204
      vendor/github.com/olivere/elastic/v7/indices_exists.go
  81. 169
      vendor/github.com/olivere/elastic/v7/indices_exists_template.go
  82. 224
      vendor/github.com/olivere/elastic/v7/indices_flush.go
  83. 281
      vendor/github.com/olivere/elastic/v7/indices_flush_synced.go
  84. 235
      vendor/github.com/olivere/elastic/v7/indices_forcemerge.go
  85. 229
      vendor/github.com/olivere/elastic/v7/indices_freeze.go
  86. 237
      vendor/github.com/olivere/elastic/v7/indices_get.go
  87. 230
      vendor/github.com/olivere/elastic/v7/indices_get_aliases.go
  88. 238
      vendor/github.com/olivere/elastic/v7/indices_get_field_mapping.go
  89. 225
      vendor/github.com/olivere/elastic/v7/indices_get_mapping.go
  90. 238
      vendor/github.com/olivere/elastic/v7/indices_get_settings.go
  91. 184
      vendor/github.com/olivere/elastic/v7/indices_get_template.go
  92. 227
      vendor/github.com/olivere/elastic/v7/indices_open.go
  93. 399
      vendor/github.com/olivere/elastic/v7/indices_put_alias.go
  94. 261
      vendor/github.com/olivere/elastic/v7/indices_put_mapping.go
  95. 242
      vendor/github.com/olivere/elastic/v7/indices_put_settings.go
  96. 259
      vendor/github.com/olivere/elastic/v7/indices_put_template.go
  97. 149
      vendor/github.com/olivere/elastic/v7/indices_refresh.go
  98. 324
      vendor/github.com/olivere/elastic/v7/indices_rollover.go
  99. 278
      vendor/github.com/olivere/elastic/v7/indices_segments.go
  100. 231
      vendor/github.com/olivere/elastic/v7/indices_shrink.go
  101. Some files were not shown because too many files have changed in this diff Show More

@ -86,6 +86,12 @@ services:
pull: default
image: gitea/test-openldap:latest
- name: elasticsearch
pull: default
environment:
discovery.type: single-node
image: elasticsearch:7.5.0
steps:
- name: fetch-tags
pull: default

@ -368,8 +368,12 @@ CONN_MAX_LIFETIME = 3s
MAX_OPEN_CONNS = 0
[indexer]
; Issue indexer type, currently support: bleve or db, default is bleve
; Issue indexer type, currently support: bleve, db or elasticsearch, default is bleve
ISSUE_INDEXER_TYPE = bleve
; Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@localhost:9200
; Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
ISSUE_INDEXER_NAME = gitea_issues
; Issue indexer storage path, available when ISSUE_INDEXER_TYPE is bleve
ISSUE_INDEXER_PATH = indexers/issues.bleve
; Issue indexer queue, currently support: channel, levelqueue or redis, default is levelqueue

@ -228,8 +228,10 @@ relation to port exhaustion.
## Indexer (`indexer`)
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently support: bleve or db, if it's db, below issue indexer item will be invalid.
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search.
- `ISSUE_INDEXER_TYPE`: **bleve**: Issue indexer type, currently supported: `bleve`, `db` or `elasticsearch`.
- `ISSUE_INDEXER_CONN_STR`: ****: Issue indexer connection string, available when ISSUE_INDEXER_TYPE is elasticsearch. i.e. http://elastic:changeme@localhost:9200
- `ISSUE_INDEXER_NAME`: **gitea_issues**: Issue indexer name, available when ISSUE_INDEXER_TYPE is elasticsearch
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: Index file used for issue search; available when ISSUE_INDEXER_TYPE is bleve and elasticsearch.
- The next 4 configuration values are deprecated and should be set in `queue.issue_indexer` however are kept for backwards compatibility:
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: Issue indexer queue, currently supports:`channel`, `levelqueue`, `redis`.
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: When `ISSUE_INDEXER_QUEUE_TYPE` is `levelqueue`, this will be the queue will be saved path.

@ -89,7 +89,9 @@ menu:
## Indexer (`indexer`)
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型,当前支持 `bleve``db`,当为 `db` 时其它工单索引项可不用设置。
- `ISSUE_INDEXER_TYPE`: **bleve**: 工单索引类型,当前支持 `bleve`, `db``elasticsearch`,当为 `db` 时其它工单索引项可不用设置。
- `ISSUE_INDEXER_CONN_STR`: ****: 工单索引连接字符串,仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。例如: http://elastic:changeme@localhost:9200
- `ISSUE_INDEXER_NAME`: **gitea_issues**: 工单索引名称,仅当 ISSUE_INDEXER_TYPE 为 `elasticsearch` 时有效。
- `ISSUE_INDEXER_PATH`: **indexers/issues.bleve**: 工单索引文件存放路径,当索引类型为 `bleve` 时有效。
- `ISSUE_INDEXER_QUEUE_TYPE`: **levelqueue**: 工单索引队列类型,当前支持 `channel``levelqueue``redis`
- `ISSUE_INDEXER_QUEUE_DIR`: **indexers/issues.queue**: 当 `ISSUE_INDEXER_QUEUE_TYPE``levelqueue` 时,保存索引队列的磁盘路径。

@ -74,6 +74,7 @@ require (
github.com/nfnt/resize v0.0.0-20160724205520-891127d8d1b5
github.com/niklasfasching/go-org v0.1.9
github.com/oliamb/cutter v0.2.2
github.com/olivere/elastic/v7 v7.0.9
github.com/pkg/errors v0.8.1
github.com/pquerna/otp v0.0.0-20160912161815-54653902c20e
github.com/prometheus/client_golang v1.1.0

@ -68,6 +68,7 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a h1:idn718Q4B6AGu/h5Sxe66HYVdqdGu2l9Iebqhi/AEoA=
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
github.com/aws/aws-sdk-go v1.25.25/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@ -154,6 +155,8 @@ github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 h1:E2s37DuLxFhQD
github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@ -248,6 +251,7 @@ github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe h1:lXe2qZdvpiX5WZ
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
@ -322,6 +326,7 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOl
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA=
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmhodges/levigo v1.0.0 h1:q5EC36kV79HWeTBWsod3mG11EgStG3qArTKcvlksN1U=
github.com/jmhodges/levigo v1.0.0/go.mod h1:Q6Qx+uH3RAqyK4rFQroq9RL7mdkABMcfhEI+nNuzMJQ=
github.com/joho/godotenv v1.3.0 h1:Zjp+RcGpHhGlrMbJzXTrZZPrWj+1vfm90La1wgB6Bhc=
@ -411,6 +416,8 @@ github.com/niklasfasching/go-org v0.1.9/go.mod h1:AsLD6X7djzRIz4/RFZu8vwRL0VGjUv
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/oliamb/cutter v0.2.2 h1:Lfwkya0HHNU1YLnGv2hTkzHfasrSMkgv4Dn+5rmlk3k=
github.com/oliamb/cutter v0.2.2/go.mod h1:4BenG2/4GuRBDbVm/OPahDVqbrOemzpPiG5mi1iryBU=
github.com/olivere/elastic/v7 v7.0.9 h1:+bTR1xJbfLYD8WnTBt9672mFlKxjfWRJpEQ1y8BMS3g=
github.com/olivere/elastic/v7 v7.0.9/go.mod h1:2TeRd0vhLRTK9zqm5xP0uLiVeZ5yUoL7kZ+8SZA9r9Y=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
@ -418,6 +425,7 @@ github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
@ -486,6 +494,7 @@ github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1
github.com/smartystreets/assertions v0.0.0-20190116191733-b6c0e53d7304/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v1.0.1 h1:voD4ITNjPL5jjBfgR/r8fPIIBrliWrWHeiJApdr3r4w=
github.com/smartystreets/assertions v1.0.1/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c/go.mod h1:XDJAKZRPZ1CvBcN2aX5YOUTYGHki24fSF0Iv48Ibg0s=
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337 h1:WN9BUFbdyOsSH/XohnWpXOlq9NBD5sGAB2FciQMUEe8=
@ -573,6 +582,7 @@ go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qL
go.opencensus.io v0.20.1/go.mod h1:6WKK9ahsWS3RSO+PY9ZHZUfv2irvY6gN279GOPZjmmk=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=

@ -10,6 +10,8 @@ PASSWD = {{TEST_MYSQL_PASSWORD}}
SSL_MODE = disable
[indexer]
ISSUE_INDEXER_TYPE = elasticsearch
ISSUE_INDEXER_CONN_STR = http://elastic:changeme@elasticsearch:9200
ISSUE_INDEXER_PATH = integrations/indexers-mysql/issues.bleve
REPO_INDEXER_ENABLED = true
REPO_INDEXER_PATH = integrations/indexers-mysql/repos.bleve

@ -170,7 +170,7 @@ func NewBleveIndexer(indexDir string) *BleveIndexer {
}
}
// Init will initial the indexer
// Init will initialize the indexer
func (b *BleveIndexer) Init() (bool, error) {
var err error
b.indexer, err = openIndexer(b.indexDir, issueIndexerLatestVersion)

@ -0,0 +1,230 @@
// Copyright 2019 The Gitea Authors. All rights reserved.
// Use of this source code is governed by a MIT-style
// license that can be found in the LICENSE file.
package issues
import (
"context"
"errors"
"fmt"
"strconv"
"time"
"code.gitea.io/gitea/modules/log"
"github.com/olivere/elastic/v7"
)
var (
_ Indexer = &ElasticSearchIndexer{}
)
// ElasticSearchIndexer implements Indexer interface
type ElasticSearchIndexer struct {
client *elastic.Client
indexerName string
}
type elasticLogger struct {
*log.Logger
}
func (l elasticLogger) Printf(format string, args ...interface{}) {
_ = l.Logger.Log(2, l.Logger.GetLevel(), format, args...)
}
// NewElasticSearchIndexer creates a new elasticsearch indexer
func NewElasticSearchIndexer(url, indexerName string) (*ElasticSearchIndexer, error) {
opts := []elastic.ClientOptionFunc{
elastic.SetURL(url),
elastic.SetSniff(false),
elastic.SetHealthcheckInterval(10 * time.Second),
elastic.SetGzip(false),
}
logger := elasticLogger{log.GetLogger(log.DEFAULT)}
if logger.GetLevel() == log.TRACE || logger.GetLevel() == log.DEBUG {
opts = append(opts, elastic.SetTraceLog(logger))
} else if logger.GetLevel() == log.ERROR || logger.GetLevel() == log.CRITICAL || logger.GetLevel() == log.FATAL {
opts = append(opts, elastic.SetErrorLog(logger))
} else if logger.GetLevel() == log.INFO || logger.GetLevel() == log.WARN {
opts = append(opts, elastic.SetInfoLog(logger))
}
client, err := elastic.NewClient(opts...)
if err != nil {
return nil, err
}
return &ElasticSearchIndexer{
client: client,
indexerName: indexerName,
}, nil
}
const (
defaultMapping = `{
"mappings": {
"properties": {
"id": {
"type": "integer",
"index": true
},
"repo_id": {
"type": "integer",
"index": true
},
"title": {
"type": "text",
"index": true
},
"content": {
"type": "text",
"index": true
},
"comments": {
"type" : "text",
"index": true
}
}
}
}`
)
// Init will initialize the indexer
func (b *ElasticSearchIndexer) Init() (bool, error) {
ctx := context.Background()
exists, err := b.client.IndexExists(b.indexerName).Do(ctx)
if err != nil {
return false, err
}
if !exists {
var mapping = defaultMapping
createIndex, err := b.client.CreateIndex(b.indexerName).BodyString(mapping).Do(ctx)
if err != nil {
return false, err
}
if !createIndex.Acknowledged {
return false, errors.New("init failed")
}
return false, nil
}
return true, nil
}
// Index will save the index data
func (b *ElasticSearchIndexer) Index(issues []*IndexerData) error {
if len(issues) == 0 {
return nil
} else if len(issues) == 1 {
issue := issues[0]
_, err := b.client.Index().
Index(b.indexerName).
Id(fmt.Sprintf("%d", issue.ID)).
BodyJson(map[string]interface{}{
"id": issue.ID,
"repo_id": issue.RepoID,
"title": issue.Title,
"content": issue.Content,
"comments": issue.Comments,
}).
Do(context.Background())
return err
}
reqs := make([]elastic.BulkableRequest, 0)
for _, issue := range issues {
reqs = append(reqs,
elastic.NewBulkIndexRequest().
Index(b.indexerName).
Id(fmt.Sprintf("%d", issue.ID)).
Doc(map[string]interface{}{
"id": issue.ID,
"repo_id": issue.RepoID,
"title": issue.Title,
"content": issue.Content,
"comments": issue.Comments,
}),
)
}
_, err := b.client.Bulk().
Index(b.indexerName).
Add(reqs...).
Do(context.Background())
return err
}
// Delete deletes indexes by ids
func (b *ElasticSearchIndexer) Delete(ids ...int64) error {
if len(ids) == 0 {
return nil
} else if len(ids) == 1 {
_, err := b.client.Delete().
Index(b.indexerName).
Id(fmt.Sprintf("%d", ids[0])).
Do(context.Background())
return err
}
reqs := make([]elastic.BulkableRequest, 0)
for _, id := range ids {
reqs = append(reqs,
elastic.NewBulkDeleteRequest().
Index(b.indexerName).
Id(fmt.Sprintf("%d", id)),
)
}
_, err := b.client.Bulk().
Index(b.indexerName).
Add(reqs...).
Do(context.Background())
return err
}
// Search searches for issues by given conditions.
// Returns the matching issue IDs
func (b *ElasticSearchIndexer) Search(keyword string, repoIDs []int64, limit, start int) (*SearchResult, error) {
kwQuery := elastic.NewMultiMatchQuery(keyword, "title", "content", "comments")
query := elastic.NewBoolQuery()
query = query.Must(kwQuery)
if len(repoIDs) > 0 {
var repoStrs = make([]interface{}, 0, len(repoIDs))
for _, repoID := range repoIDs {
repoStrs = append(repoStrs, repoID)
}
repoQuery := elastic.NewTermsQuery("repo_id", repoStrs...)
query = query.Must(repoQuery)
}
searchResult, err := b.client.Search().
Index(b.indexerName).
Query(query).
Sort("id", true).
From(start).Size(limit).
Do(context.Background())
if err != nil {
return nil, err
}
hits := make([]Match, 0, limit)
for _, hit := range searchResult.Hits.Hits {
id, _ := strconv.ParseInt(hit.Id, 10, 64)
hits = append(hits, Match{
ID: id,
})
}
return &SearchResult{
Total: searchResult.TotalHits(),
Hits: hits,
}, nil
}
// Close implements indexer
func (b *ElasticSearchIndexer) Close() {}

@ -21,13 +21,13 @@ import (
// IndexerData data stored in the issue indexer
type IndexerData struct {
ID int64
RepoID int64
Title string
Content string
Comments []string
IsDelete bool
IDs []int64
ID int64 `json:"id"`
RepoID int64 `json:"repo_id"`
Title string `json:"title"`
Content string `json:"content"`
Comments []string `json:"comments"`
IsDelete bool `json:"is_delete"`
IDs []int64 `json:"ids"`
}
// Match represents on search result
@ -100,7 +100,7 @@ func InitIssueIndexer(syncReindex bool) {
// Create the Queue
switch setting.Indexer.IssueType {
case "bleve":
case "bleve", "elasticsearch":
handler := func(data ...queue.Data) {
indexer := holder.get()
if indexer == nil {
@ -160,6 +160,19 @@ func InitIssueIndexer(syncReindex bool) {
log.Info("PID: %d Issue Indexer closed", os.Getpid())
})
log.Debug("Created Bleve Indexer")
case "elasticsearch":
graceful.GetManager().RunWithShutdownFns(func(_, atTerminate func(context.Context, func())) {
issueIndexer, err := NewElasticSearchIndexer(setting.Indexer.IssueConnStr, "gitea_issues")
if err != nil {
log.Fatal("Unable to initialize Elastic Search Issue Indexer: %v", err)
}
exist, err := issueIndexer.Init()
if err != nil {
log.Fatal("Unable to issueIndexer.Init: %v", err)
}
populate = !exist
holder.set(issueIndexer)
})
case "db":
issueIndexer := &DBIndexer{}
holder.set(issueIndexer)
@ -308,6 +321,7 @@ func DeleteRepoIssueIndexer(repo *models.Repository) {
}
// SearchIssuesByKeyword search issue ids by keywords and repo id
// WARNNING: You have to ensure user have permission to visit repoIDs' issues
func SearchIssuesByKeyword(repoIDs []int64, keyword string) ([]int64, error) {
var issueIDs []int64
indexer := holder.get()
@ -316,7 +330,7 @@ func SearchIssuesByKeyword(repoIDs []int64, keyword string) ([]int64, error) {
log.Error("SearchIssuesByKeyword(): unable to get indexer!")
return nil, fmt.Errorf("unable to get issue indexer")
}
res, err := indexer.Search(keyword, repoIDs, 1000, 0)
res, err := indexer.Search(keyword, repoIDs, 50, 0)
if err != nil {
return nil, err
}

@ -27,20 +27,25 @@ var (
Indexer = struct {
IssueType string
IssuePath string
RepoIndexerEnabled bool
RepoPath string
UpdateQueueLength int
MaxIndexerFileSize int64
IssueConnStr string
IssueIndexerName string
IssueQueueType string
IssueQueueDir string
IssueQueueConnStr string
IssueQueueBatchNumber int
StartupTimeout time.Duration
IncludePatterns []glob.Glob
ExcludePatterns []glob.Glob
RepoIndexerEnabled bool
RepoPath string
UpdateQueueLength int
MaxIndexerFileSize int64
IncludePatterns []glob.Glob
ExcludePatterns []glob.Glob
}{
IssueType: "bleve",
IssuePath: "indexers/issues.bleve",
IssueConnStr: "",
IssueIndexerName: "gitea_issues",
IssueQueueType: LevelQueueType,
IssueQueueDir: "indexers/issues.queue",
IssueQueueConnStr: "",
@ -57,6 +62,14 @@ func newIndexerService() {
if !filepath.IsAbs(Indexer.IssuePath) {
Indexer.IssuePath = path.Join(AppWorkPath, Indexer.IssuePath)
}
Indexer.IssueConnStr = sec.Key("ISSUE_INDEXER_CONN_STR").MustString(Indexer.IssueConnStr)
Indexer.IssueIndexerName = sec.Key("ISSUE_INDEXER_NAME").MustString(Indexer.IssueIndexerName)
Indexer.IssueQueueType = sec.Key("ISSUE_INDEXER_QUEUE_TYPE").MustString(LevelQueueType)
Indexer.IssueQueueDir = sec.Key("ISSUE_INDEXER_QUEUE_DIR").MustString(path.Join(AppDataPath, "indexers/issues.queue"))
Indexer.IssueQueueConnStr = sec.Key("ISSUE_INDEXER_QUEUE_CONN_STR").MustString(path.Join(AppDataPath, ""))
Indexer.IssueQueueBatchNumber = sec.Key("ISSUE_INDEXER_QUEUE_BATCH_NUMBER").MustInt(20)
Indexer.RepoIndexerEnabled = sec.Key("REPO_INDEXER_ENABLED").MustBool(false)
Indexer.RepoPath = sec.Key("REPO_INDEXER_PATH").MustString(path.Join(AppDataPath, "indexers/repos.bleve"))
if !filepath.IsAbs(Indexer.RepoPath) {
@ -64,13 +77,8 @@ func newIndexerService() {
}
Indexer.IncludePatterns = IndexerGlobFromString(sec.Key("REPO_INDEXER_INCLUDE").MustString(""))
Indexer.ExcludePatterns = IndexerGlobFromString(sec.Key("REPO_INDEXER_EXCLUDE").MustString(""))
Indexer.UpdateQueueLength = sec.Key("UPDATE_BUFFER_LEN").MustInt(20)
Indexer.MaxIndexerFileSize = sec.Key("MAX_FILE_SIZE").MustInt64(1024 * 1024)
Indexer.IssueQueueType = sec.Key("ISSUE_INDEXER_QUEUE_TYPE").MustString(LevelQueueType)
Indexer.IssueQueueDir = sec.Key("ISSUE_INDEXER_QUEUE_DIR").MustString(path.Join(AppDataPath, "indexers/issues.queue"))
Indexer.IssueQueueConnStr = sec.Key("ISSUE_INDEXER_QUEUE_CONN_STR").MustString(path.Join(AppDataPath, ""))
Indexer.IssueQueueBatchNumber = sec.Key("ISSUE_INDEXER_QUEUE_BATCH_NUMBER").MustInt(20)
Indexer.StartupTimeout = sec.Key("STARTUP_TIMEOUT").MustDuration(30 * time.Second)
}

@ -0,0 +1,5 @@
.root
*_easyjson.go
*.iml
.idea
*.swp

@ -0,0 +1,12 @@
language: go
go:
- tip
- stable
matrix:
allow_failures:
- go: tip
install:
- go get golang.org/x/lint/golint

@ -0,0 +1,52 @@
all: test
clean:
rm -rf bin
rm -rf tests/*_easyjson.go
rm -rf benchmark/*_easyjson.go
build:
go build -i -o ./bin/easyjson ./easyjson
generate: build
bin/easyjson -stubs \
./tests/snake.go \
./tests/data.go \
./tests/omitempty.go \
./tests/nothing.go \
./tests/named_type.go \
./tests/custom_map_key_type.go \
./tests/embedded_type.go \
./tests/reference_to_pointer.go \
bin/easyjson -all ./tests/data.go
bin/easyjson -all ./tests/nothing.go
bin/easyjson -all ./tests/errors.go
bin/easyjson -snake_case ./tests/snake.go
bin/easyjson -omit_empty ./tests/omitempty.go
bin/easyjson -build_tags=use_easyjson ./benchmark/data.go
bin/easyjson ./tests/nested_easy.go
bin/easyjson ./tests/named_type.go
bin/easyjson ./tests/custom_map_key_type.go
bin/easyjson ./tests/embedded_type.go
bin/easyjson ./tests/reference_to_pointer.go
bin/easyjson ./tests/key_marshaler_map.go
bin/easyjson -disallow_unknown_fields ./tests/disallow_unknown.go
test: generate
go test \
./tests \
./jlexer \
./gen \
./buffer
cd benchmark && go test -benchmem -tags use_easyjson -bench .
golint -set_exit_status ./tests/*_easyjson.go
bench-other: generate
cd benchmark && make
bench-python:
benchmark/ujson.sh
.PHONY: clean generate test build

@ -0,0 +1,336 @@
# easyjson [![Build Status](https://travis-ci.org/mailru/easyjson.svg?branch=master)](https://travis-ci.org/mailru/easyjson) [![Go Report Card](https://goreportcard.com/badge/github.com/mailru/easyjson)](https://goreportcard.com/report/github.com/mailru/easyjson)
Package easyjson provides a fast and easy way to marshal/unmarshal Go structs
to/from JSON without the use of reflection. In performance tests, easyjson
outperforms the standard `encoding/json` package by a factor of 4-5x, and other
JSON encoding packages by a factor of 2-3x.
easyjson aims to keep generated Go code simple enough so that it can be easily
optimized or fixed. Another goal is to provide users with the ability to
customize the generated code by providing options not available with the
standard `encoding/json` package, such as generating "snake_case" names or
enabling `omitempty` behavior by default.
## Usage
```sh
# install
go get -u github.com/mailru/easyjson/...
# run
easyjson -all <file>.go
```
The above will generate `<file>_easyjson.go` containing the appropriate marshaler and
unmarshaler funcs for all structs contained in `<file>.go`.
Please note that easyjson requires a full Go build environment and the `GOPATH`
environment variable to be set. This is because easyjson code generation
invokes `go run` on a temporary file (an approach to code generation borrowed
from [ffjson](https://github.com/pquerna/ffjson)).
## Options
```txt
Usage of easyjson:
-all
generate marshaler/unmarshalers for all structs in a file
-build_tags string
build tags to add to generated file
-leave_temps
do not delete temporary files
-no_std_marshalers
don't generate MarshalJSON/UnmarshalJSON funcs
-noformat
do not run 'gofmt -w' on output file
-omit_empty
omit empty fields by default
-output_filename string
specify the filename of the output
-pkg
process the whole package instead of just the given file
-snake_case
use snake_case names instead of CamelCase by default
-lower_camel_case
use lowerCamelCase instead of CamelCase by default
-stubs
only generate stubs for marshaler/unmarshaler funcs
-disallow_unknown_fields
return error if some unknown field in json appeared
```
Using `-all` will generate marshalers/unmarshalers for all Go structs in the
file. If `-all` is not provided, then only those structs whose preceding
comment starts with `easyjson:json` will have marshalers/unmarshalers
generated. For example:
```go
//easyjson:json
type A struct {}
```
Additional option notes:
* `-snake_case` tells easyjson to generate snake\_case field names by default
(unless overridden by a field tag). The CamelCase to snake\_case conversion
algorithm should work in most cases (ie, HTTPVersion will be converted to
"http_version").
* `-build_tags` will add the specified build tags to generated Go sources.
## Generated Marshaler/Unmarshaler Funcs
For Go struct types, easyjson generates the funcs `MarshalEasyJSON` /
`UnmarshalEasyJSON` for marshaling/unmarshaling JSON. In turn, these satisify
the `easyjson.Marshaler` and `easyjson.Unmarshaler` interfaces and when used in
conjunction with `easyjson.Marshal` / `easyjson.Unmarshal` avoid unnecessary
reflection / type assertions during marshaling/unmarshaling to/from JSON for Go
structs.
easyjson also generates `MarshalJSON` and `UnmarshalJSON` funcs for Go struct
types compatible with the standard `json.Marshaler` and `json.Unmarshaler`
interfaces. Please be aware that using the standard `json.Marshal` /
`json.Unmarshal` for marshaling/unmarshaling will incur a significant
performance penalty when compared to using `easyjson.Marshal` /
`easyjson.Unmarshal`.
Additionally, easyjson exposes utility funcs that use the `MarshalEasyJSON` and
`UnmarshalEasyJSON` for marshaling/unmarshaling to and from standard readers
and writers. For example, easyjson provides `easyjson.MarshalToHTTPResponseWriter`
which marshals to the standard `http.ResponseWriter`. Please see the [GoDoc
listing](https://godoc.org/github.com/mailru/easyjson) for the full listing of
utility funcs that are available.
## Controlling easyjson Marshaling and Unmarshaling Behavior
Go types can provide their own `MarshalEasyJSON` and `UnmarshalEasyJSON` funcs
that satisify the `easyjson.Marshaler` / `easyjson.Unmarshaler` interfaces.
These will be used by `easyjson.Marshal` and `easyjson.Unmarshal` when defined
for a Go type.
Go types can also satisify the `easyjson.Optional` interface, which allows the
type to define its own `omitempty` logic.
## Type Wrappers
easyjson provides additional type wrappers defined in the `easyjson/opt`
package. These wrap the standard Go primitives and in turn satisify the
easyjson interfaces.
The `easyjson/opt` type wrappers are useful when needing to distinguish between
a missing value and/or when needing to specifying a default value. Type
wrappers allow easyjson to avoid additional pointers and heap allocations and
can significantly increase performance when used properly.
## Memory Pooling
easyjson uses a buffer pool that allocates data in increasing chunks from 128
to 32768 bytes. Chunks of 512 bytes and larger will be reused with the help of
`sync.Pool`. The maximum size of a chunk is bounded to reduce redundant memory
allocation and to allow larger reusable buffers.
easyjson's custom allocation buffer pool is defined in the `easyjson/buffer`
package, and the default behavior pool behavior can be modified (if necessary)
through a call to `buffer.Init()` prior to any marshaling or unmarshaling.
Please see the [GoDoc listing](https://godoc.org/github.com/mailru/easyjson/buffer)
for more information.
## Issues, Notes, and Limitations
* easyjson is still early in its development. As such, there are likely to be
bugs and missing features when compared to `encoding/json`. In the case of a
missing feature or bug, please create a GitHub issue. Pull requests are
welcome!
* Unlike `encoding/json`, object keys are case-sensitive. Case-insensitive
matching is not currently provided due to the significant performance hit
when doing case-insensitive key matching. In the future, case-insensitive
object key matching may be provided via an option to the generator.
* easyjson makes use of `unsafe`, which simplifies the code and
provides significant performance benefits by allowing no-copy
conversion from `[]byte` to `string`. That said, `unsafe` is used
only when unmarshaling and parsing JSON, and any `unsafe` operations
/ memory allocations done will be safely deallocated by
easyjson. Set the build tag `easyjson_nounsafe` to compile it
without `unsafe`.
* easyjson is compatible with Google App Engine. The `appengine` build
tag (set by App Engine's environment) will automatically disable the
use of `unsafe`, which is not allowed in App Engine's Standard
Environment. Note that the use with App Engine is still experimental.
* Floats are formatted using the default precision from Go's `strconv` package.
As such, easyjson will not correctly handle high precision floats when
marshaling/unmarshaling JSON. Note, however, that there are very few/limited
uses where this behavior is not sufficient for general use. That said, a
different package may be needed if precise marshaling/unmarshaling of high
precision floats to/from JSON is required.
* While unmarshaling, the JSON parser does the minimal amount of work needed to
skip over unmatching parens, and as such full validation is not done for the
entire JSON value being unmarshaled/parsed.
* Currently there is no true streaming support for encoding/decoding as
typically for many uses/protocols the final, marshaled length of the JSON
needs to be known prior to sending the data. Currently this is not possible
with easyjson's architecture.
* easyjson parser and codegen based on reflection, so it wont works on `package main`
files, because they cant be imported by parser.
## Benchmarks
Most benchmarks were done using the example
[13kB example JSON](https://dev.twitter.com/rest/reference/get/search/tweets)
(9k after eliminating whitespace). This example is similar to real-world data,
is well-structured, and contains a healthy variety of different types, making
it ideal for JSON serialization benchmarks.
Note:
* For small request benchmarks, an 80 byte portion of the above example was
used.
* For large request marshaling benchmarks, a struct containing 50 regular
samples was used, making a ~500kB output JSON.
* Benchmarks are showing the results of easyjson's default behaviour,
which makes use of `unsafe`.
Benchmarks are available in the repository and can be run by invoking `make`.
### easyjson vs. encoding/json
easyjson is roughly 5-6 times faster than the standard `encoding/json` for
unmarshaling, and 3-4 times faster for non-concurrent marshaling. Concurrent
marshaling is 6-7x faster if marshaling to a writer.
### easyjson vs. ffjson
easyjson uses the same approach for JSON marshaling as
[ffjson](https://github.com/pquerna/ffjson), but takes a significantly
different approach to lexing and parsing JSON during unmarshaling. This means
easyjson is roughly 2-3x faster for unmarshaling and 1.5-2x faster for
non-concurrent unmarshaling.
As of this writing, `ffjson` seems to have issues when used concurrently:
specifically, large request pooling hurts `ffjson`'s performance and causes
scalability issues. These issues with `ffjson` can likely be fixed, but as of
writing remain outstanding/known issues with `ffjson`.
easyjson and `ffjson` have similar performance for small requests, however
easyjson outperforms `ffjson` by roughly 2-5x times for large requests when
used with a writer.
### easyjson vs. go/codec
[go/codec](https://github.com/ugorji/go) provides
compile-time helpers for JSON generation. In this case, helpers do not work
like marshalers as they are encoding-independent.
easyjson is generally 2x faster than `go/codec` for non-concurrent benchmarks
and about 3x faster for concurrent encoding (without marshaling to a writer).
In an attempt to measure marshaling performance of `go/codec` (as opposed to
allocations/memcpy/writer interface invocations), a benchmark was done with
resetting length of a byte slice rather than resetting the whole slice to nil.
However, the optimization in this exact form may not be applicable in practice,
since the memory is not freed between marshaling operations.
### easyjson vs 'ujson' python module
[ujson](https://github.com/esnme/ultrajson) is using C code for parsing, so it
is interesting to see how plain golang compares to that. It is imporant to note
that the resulting object for python is slower to access, since the library
parses JSON object into dictionaries.
easyjson is slightly faster for unmarshaling and 2-3x faster than `ujson` for
marshaling.
### Benchmark Results
`ffjson` results are from February 4th, 2016, using the latest `ffjson` and go1.6.
`go/codec` results are from March 4th, 2016, using the latest `go/codec` and go1.6.
#### Unmarshaling
| lib | json size | MB/s | allocs/op | B/op |
|:---------|:----------|-----:|----------:|------:|
| standard | regular | 22 | 218 | 10229 |
| standard | small | 9.7 | 14 | 720 |
| | | | | |
| easyjson | regular | 125 | 128 | 9794 |
| easyjson | small | 67 | 3 | 128 |
| | | | | |
| ffjson | regular | 66 | 141 | 9985 |
| ffjson | small | 17.6 | 10 | 488 |
| | | | | |
| codec | regular | 55 | 434 | 19299 |
| codec | small | 29 | 7 | 336 |
| | | | | |
| ujson | regular | 103 | N/A | N/A |
#### Marshaling, one goroutine.
| lib | json size | MB/s | allocs/op | B/op |
|:----------|:----------|-----:|----------:|------:|
| standard | regular | 75 | 9 | 23256 |
| standard | small | 32 | 3 | 328 |
| standard | large | 80 | 17 | 1.2M |
| | | | | |
| easyjson | regular | 213 | 9 | 10260 |
| easyjson* | regular | 263 | 8 | 742 |
| easyjson | small | 125 | 1 | 128 |
| easyjson | large | 212 | 33 | 490k |
| easyjson* | large | 262 | 25 | 2879 |
| | | | | |
| ffjson | regular | 122 | 153 | 21340 |
| ffjson** | regular | 146 | 152 | 4897 |
| ffjson | small | 36 | 5 | 384 |
| ffjson** | small | 64 | 4 | 128 |
| ffjson | large | 134 | 7317 | 818k |
| ffjson** | large | 125 | 7320 | 827k |
| | | | | |
| codec | regular | 80 | 17 | 33601 |
| codec*** | regular | 108 | 9 | 1153 |
| codec | small | 42 | 3 | 304 |
| codec*** | small | 56 | 1 | 48 |
| codec | large | 73 | 483 | 2.5M |
| codec*** | large | 103 | 451 | 66007 |
| | | | | |
| ujson | regular | 92 | N/A | N/A |
\* marshaling to a writer,
\*\* using `ffjson.Pool()`,
\*\*\* reusing output slice instead of resetting it to nil
#### Marshaling, concurrent.
| lib | json size | MB/s | allocs/op | B/op |
|:----------|:----------|-----:|----------:|------:|
| standard | regular | 252 | 9 | 23257 |
| standard | small | 124 | 3 | 328 |
| standard | large | 289 | 17 | 1.2M |
| | | | | |
| easyjson | regular | 792 | 9 | 10597 |
| easyjson* | regular | 1748 | 8 | 779 |
| easyjson | small | 333 | 1 | 128 |
| easyjson | large | 718 | 36 | 548k |
| easyjson* | large | 2134 | 25 | 4957 |
| | | | | |
| ffjson | regular | 301 | 153 | 21629 |
| ffjson** | regular | 707 | 152 | 5148 |
| ffjson | small | 62 | 5 | 384 |
| ffjson** | small | 282 | 4 | 128 |
| ffjson | large | 438 | 7330 | 1.0M |
| ffjson** | large | 131 | 7319 | 820k |
| | | | | |
| codec | regular | 183 | 17 | 33603 |
| codec*** | regular | 671 | 9 | 1157 |
| codec | small | 147 | 3 | 304 |
| codec*** | small | 299 | 1 | 48 |
| codec | large | 190 | 483 | 2.5M |
| codec*** | large | 752 | 451 | 77574 |
\* marshaling to a writer,
\*\* using `ffjson.Pool()`,
\*\*\* reusing output slice instead of resetting it to nil

@ -0,0 +1,3 @@
module github.com/mailru/easyjson
go 1.12

@ -0,0 +1,78 @@
// Package easyjson contains marshaler/unmarshaler interfaces and helper functions.
package easyjson
import (
"io"
"io/ioutil"
"net/http"
"strconv"
"github.com/mailru/easyjson/jlexer"
"github.com/mailru/easyjson/jwriter"
)
// Marshaler is an easyjson-compatible marshaler interface.
type Marshaler interface {
MarshalEasyJSON(w *jwriter.Writer)
}
// Marshaler is an easyjson-compatible unmarshaler interface.
type Unmarshaler interface {
UnmarshalEasyJSON(w *jlexer.Lexer)
}
// Optional defines an undefined-test method for a type to integrate with 'omitempty' logic.
type Optional interface {
IsDefined() bool
}
// Marshal returns data as a single byte slice. Method is suboptimal as the data is likely to be copied
// from a chain of smaller chunks.
func Marshal(v Marshaler) ([]byte, error) {
w := jwriter.Writer{}
v.MarshalEasyJSON(&w)
return w.BuildBytes()
}
// MarshalToWriter marshals the data to an io.Writer.
func MarshalToWriter(v Marshaler, w io.Writer) (written int, err error) {
jw := jwriter.Writer{}
v.MarshalEasyJSON(&jw)
return jw.DumpTo(w)
}
// MarshalToHTTPResponseWriter sets Content-Length and Content-Type headers for the
// http.ResponseWriter, and send the data to the writer. started will be equal to
// false if an error occurred before any http.ResponseWriter methods were actually
// invoked (in this case a 500 reply is possible).
func MarshalToHTTPResponseWriter(v Marshaler, w http.ResponseWriter) (started bool, written int, err error) {
jw := jwriter.Writer{}
v.MarshalEasyJSON(&jw)
if jw.Error != nil {
return false, 0, jw.Error
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Content-Length", strconv.Itoa(jw.Size()))
started = true
written, err = jw.DumpTo(w)
return
}
// Unmarshal decodes the JSON in data into the object.
func Unmarshal(data []byte, v Unmarshaler) error {
l := jlexer.Lexer{Data: data}
v.UnmarshalEasyJSON(&l)
return l.Error()
}
// UnmarshalFromReader reads all the data in the reader and decodes as JSON into the object.
func UnmarshalFromReader(r io.Reader, v Unmarshaler) error {
data, err := ioutil.ReadAll(r)
if err != nil {
return err
}
l := jlexer.Lexer{Data: data}
v.UnmarshalEasyJSON(&l)
return l.Error()
}

@ -0,0 +1,45 @@
package easyjson
import (
"github.com/mailru/easyjson/jlexer"
"github.com/mailru/easyjson/jwriter"
)
// RawMessage is a raw piece of JSON (number, string, bool, object, array or
// null) that is extracted without parsing and output as is during marshaling.
type RawMessage []byte
// MarshalEasyJSON does JSON marshaling using easyjson interface.
func (v *RawMessage) MarshalEasyJSON(w *jwriter.Writer) {
if len(*v) == 0 {
w.RawString("null")
} else {
w.Raw(*v, nil)
}
}
// UnmarshalEasyJSON does JSON unmarshaling using easyjson interface.
func (v *RawMessage) UnmarshalEasyJSON(l *jlexer.Lexer) {
*v = RawMessage(l.Raw())
}
// UnmarshalJSON implements encoding/json.Unmarshaler interface.
func (v *RawMessage) UnmarshalJSON(data []byte) error {
*v = data
return nil
}
var nullBytes = []byte("null")
// MarshalJSON implements encoding/json.Marshaler interface.
func (v RawMessage) MarshalJSON() ([]byte, error) {
if len(v) == 0 {
return nullBytes, nil
}
return v, nil
}
// IsDefined is required for integration with omitempty easyjson logic.
func (v *RawMessage) IsDefined() bool {
return len(*v) > 0
}

@ -0,0 +1,38 @@
# Generated by FOSSA CLI (https://github.com/fossas/fossa-cli)
# Visit https://fossa.io to learn more
version: 1
cli:
server: https://app.fossa.io
fetcher: git
project: git@github.com:olivere/elastic.git
analyze:
modules:
- name: github.com/olivere/elastic
path: .
target: github.com/olivere/elastic
type: go
- name: github.com/olivere/elastic/config
path: ./config
target: github.com/olivere/elastic/config
type: go
- name: github.com/olivere/elastic/uritemplates
path: ./uritemplates
target: github.com/olivere/elastic/uritemplates
type: go
- name: github.com/olivere/elastic/trace/opencensus
path: ./trace/opencensus
target: github.com/olivere/elastic/trace/opencensus
type: go
- name: github.com/olivere/elastic/trace/opentracing
path: ./trace/opentracing
target: github.com/olivere/elastic/trace/opentracing
type: go
- name: github.com/olivere/elastic/aws
path: ./aws
target: github.com/olivere/elastic/aws
type: go
- name: github.com/olivere/elastic/aws/v4
path: ./aws/v4
target: github.com/olivere/elastic/aws/v4
type: go

@ -0,0 +1,35 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
/.vscode/
/.idea/
/debug.test
/generator
/cluster-test/cluster-test
/cluster-test/*.log
/cluster-test/es-chaos-monkey
/go.sum
/spec
/tmp
/CHANGELOG-3.0.html

@ -0,0 +1,32 @@
sudo: required
language: go
go:
- "1.12.x"
- "1.13.x"
- tip
matrix:
allow_failures:
- go: tip
env:
- GO111MODULE=on
- GO111MODULE=off
addons:
apt:
update: true
packages:
- docker-ce
services:
- docker
before_install:
- if [[ "$TRAVIS_OS_NAME" == "linux" && ! $(which nc) ]] ; then sudo apt-get install -y netcat ; fi
- sudo sysctl -w vm.max_map_count=262144
- docker-compose pull
- docker-compose up -d
- go get -u github.com/google/go-cmp/cmp
- go get -u github.com/fortytw2/leaktest
- go get . ./aws/... ./config/... ./trace/... ./uritemplates/...
- while ! nc -z localhost 9200; do sleep 1; done
- while ! nc -z localhost 9210; do sleep 1; done
install: true # ignore the go get -t -v ./...
script:
- go test -race -deprecations -strict-decoder -v . ./aws/... ./config/... ./trace/... ./uritemplates/...

@ -0,0 +1,363 @@
# Elastic 3.0
Elasticsearch 2.0 comes with some [breaking changes](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking-changes-2.0.html). You will probably need to upgrade your application and/or rewrite part of it due to those changes.
We use that window of opportunity to also update Elastic (the Go client) from version 2.0 to 3.0. This will introduce both changes due to the Elasticsearch 2.0 update as well as changes that make Elastic cleaner by removing some old cruft.
So, to summarize:
1. Elastic 2.0 is compatible with Elasticsearch 1.7+ and is still actively maintained.
2. Elastic 3.0 is compatible with Elasticsearch 2.0+ and will soon become the new master branch.
The rest of the document is a list of all changes in Elastic 3.0.
## Pointer types
All types have changed to be pointer types, not value types. This not only is cleaner but also simplifies the API as illustrated by the following example:
Example for Elastic 2.0 (old):
```go
q := elastic.NewMatchAllQuery()
res, err := elastic.Search("one").Query(&q).Do() // notice the & here
```
Example for Elastic 3.0 (new):
```go
q := elastic.NewMatchAllQuery()
res, err := elastic.Search("one").Query(q).Do() // no more &
// ... which can be simplified as:
res, err := elastic.Search("one").Query(elastic.NewMatchAllQuery()).Do()
```
It also helps to prevent [subtle issues](https://github.com/olivere/elastic/issues/115#issuecomment-130753046).
## Query/filter merge
One of the biggest changes in Elasticsearch 2.0 is the [merge of queries and filters](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_query_dsl_changes.html#_queries_and_filters_merged). In Elasticsearch 1.x, you had a whole range of queries and filters that were basically identical (e.g. `term_query` and `term_filter`).
The practical aspect of the merge is that you can now basically use queries where once you had to use filters instead. For Elastic 3.0 this means: We could remove a whole bunch of files. Yay!
Notice that some methods still come by "filter", e.g. `PostFilter`. However, they accept a `Query` now when they used to accept a `Filter` before.
Example for Elastic 2.0 (old):
```go
q := elastic.NewMatchAllQuery()
f := elastic.NewTermFilter("tag", "important")
res, err := elastic.Search().Index("one").Query(&q).PostFilter(f)
```
Example for Elastic 3.0 (new):
```go
q := elastic.NewMatchAllQuery()
f := elastic.NewTermQuery("tag", "important") // it's a query now!
res, err := elastic.Search().Index("one").Query(q).PostFilter(f)
```
## Facets are removed
[Facets have been removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_removed_features.html#_facets_have_been_removed) in Elasticsearch 2.0. You need to use aggregations now.
## Errors
Elasticsearch 2.0 returns more information about an error in the HTTP response body. Elastic 3.0 now reads this information and makes it accessible by the consumer.
Errors and all its details are now returned in [`Error`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59).
### HTTP Status 404 (Not Found)
When Elasticsearch does not find an entity or an index, it generally returns HTTP status code 404. In Elastic 2.0 this was a valid result and didn't raise an error from the `Do` functions. This has now changed in Elastic 3.0.
Starting with Elastic 3.0, there are only two types of responses considered successful. First, responses with HTTP status codes [200..299]. Second, HEAD requests which return HTTP status 404. The latter is used by Elasticsearch to e.g. check for existence of indices or documents. All other responses will return an error.
To check for HTTP Status 404 (with non-HEAD requests), e.g. when trying to get or delete a missing document, you can use the [`IsNotFound`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L84) helper (see below).
The following example illustrates how to check for a missing document in Elastic 2.0 and what has changed in 3.0.
Example for Elastic 2.0 (old):
```go
res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
if err != nil {
// Something else went wrong (but 404 is NOT an error in Elastic 2.0)
}
if !res.Found {
// Document has not been found
}
```
Example for Elastic 3.0 (new):
```go
res, err = client.Get().Index("one").Type("tweet").Id("no-such-id").Do()
if err != nil {
if elastic.IsNotFound(err) {
// Document has not been found
} else {
// Something else went wrong
}
}
```
### HTTP Status 408 (Timeouts)
Elasticsearch now responds with HTTP status code 408 (Timeout) when a request fails due to a timeout. E.g. if you specify a timeout with the Cluster Health API, the HTTP response status will be 408 if the timeout is raised. See [here](https://github.com/elastic/elasticsearch/commit/fe3179d9cccb569784434b2135ca9ae13d5158d3) for the specific commit to the Cluster Health API.
To check for HTTP Status 408, we introduced the [`IsTimeout`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L101) helper.
Example for Elastic 2.0 (old):
```go
health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
if err != nil {
// ...
}
if health.TimedOut {
// We have a timeout
}
```
Example for Elastic 3.0 (new):
```go
health, err := client.ClusterHealth().WaitForStatus("yellow").Timeout("1s").Do()
if elastic.IsTimeout(err) {
// We have a timeout
}
```
### Bulk Errors
The error response of a bulk operation used to be a simple string in Elasticsearch 1.x.
In Elasticsearch 2.0, it returns a structured JSON object with a lot more details about the error.
These errors are now captured in an object of type [`ErrorDetails`](https://github.com/olivere/elastic/blob/release-branch.v3/errors.go#L59) which is used in [`BulkResponseItem`](https://github.com/olivere/elastic/blob/release-branch.v3/bulk.go#L206).
### Removed specific Elastic errors
The specific error types `ErrMissingIndex`, `ErrMissingType`, and `ErrMissingId` have been removed. They were only used by `DeleteService` and are replaced by a generic error message.
## Numeric types
Elastic 3.0 has settled to use `float64` everywhere. It used to be a mix of `float32` and `float64` in Elastic 2.0. E.g. all boostable queries in Elastic 3.0 now have a boost type of `float64` where it used to be `float32`.
## Pluralization
Some services accept zero, one or more indices or types to operate on.
E.g. in the `SearchService` accepts a list of zero, one, or more indices to
search and therefor had a func called `Index(index string)` and a func
called `Indices(indices ...string)`.
Elastic 3.0 now only uses the singular form that, when applicable, accepts a
variadic type. E.g. in the case of the `SearchService`, you now only have
one func with the following signature: `Index(indices ...string)`.
Notice this is only limited to `Index(...)` and `Type(...)`. There are other
services with variadic functions. These have not been changed.
## Multiple calls to variadic functions
Some services with variadic functions have cleared the underlying slice when
called while other services just add to the existing slice. This has now been
normalized to always add to the underlying slice.
Example for Elastic 2.0 (old):
```go
// Would only cleared scroll id "two"
// because ScrollId cleared the values when called multiple times
client.ClearScroll().ScrollId("one").ScrollId("two").Do()
```
Example for Elastic 3.0 (new):
```go
// Now (correctly) clears both scroll id "one" and "two"
// because ScrollId no longer clears the values when called multiple times
client.ClearScroll().ScrollId("one").ScrollId("two").Do()
```
## Ping service requires URL
The `Ping` service raised some issues because it is different from all
other services. If not explicitly given a URL, it always pings `127.0.0.1:9200`.
Users expected to ping the cluster, but that is not possible as the cluster
can be a set of many nodes: So which node do we ping then?
To make it more clear, the `Ping` function on the client now requires users
to explicitly set the URL of the node to ping.
## Meta fields
Many of the meta fields e.g. `_parent` or `_routing` are now
[part of the top-level of a document](https://www.elastic.co/guide/en/elasticsearch/reference/2.0/_mapping_changes.html#migration-meta-fields)
and are no longer returned as parts of the `fields` object. We had to change
larger parts of e.g. the `Reindexer` to get it to work seamlessly with Elasticsearch 2.0.