ES Merge multiple indices in one by reindex API
目錄
如何透過 ES Reindex API 以及 alias 搭配不停機 indices 合併?
首先，確保你的環境寫入是針對 Alias Target 做寫入，這樣就可以在後面隨便你搞不停機。
流程簡述
- 讓 Alias 指向一個全新的 indices 做線上串流寫入
- 建立新的 indices 作為你要合併的 Destination
- 透過 reindex 進行多 indices 合併，同時如何進行加速
- 檢查 reindex 進行的狀態
- 完成後把為了加速的設定改回來(設定 Replica 會造成 YELLOW Status)
1. Check the new version indices with right provided name
GET _cat/indices?v&s=index
GET _cat/indices/qlog-fff*?v&s=index2. Check the index and aliaes create or not
GET _cat/aliases/qlog-fff*?v&s=alias
GET _alias3. Change is_write_index pointer from the old to the new one.
POST /_aliases
{
"actions" : [
{
"add" : {
"index" : "qlog-fff-202104-1",
"alias" : "qlog-fff-202104",
"is_write_index" : false
}
},
{
"add" : {
"index" : "qlog-fff-202104-0408-1",
"alias" : "qlog-fff-202104",
"is_write_index" : true
}
}
]
}4. Create the new index for reindex merge target
PUT qlog-f-vbs-202104-0000-1/
{
"settings": {
"index": {
"number_of_shards": 2, // Change here depends on your case
"number_of_replicas": 0,
"refresh_interval": "-1"
}
}
}
Speed Up Reindex
- set the
number_of_replicasto 0 - set
refresh_intervalto -1 - More, check the number_of_shards case by case (calculate the total indices after merging)
5. Do reindex for merging indices
POST _reindex?wait_for_completion=false&slices=auto&refresh=false
{
"source": {
"index": ["qlog-fff-202105-000002", "qlog-fff-202104-1"],
"size": 5000
},
"dest": {
"index": "qlog-fff-202104-0000-1"
}
}
Speed Up Reindex
- Set the batch size
- use sliceing action to run reindex with multiple threads
6. Check reindex status
GET _tasks?detailed=true&actions=*reindex7. Recover the settings for the new indcies
GET qlog-fff-202104-0000-1/_settings
PUT qlog-fff-202104-0000-1/_settings
{
"index": {
"refresh_interval": null,
"number_of_replicas": 1
}
}After reindex action, set the
refresh_intervalandreplicaback
8. Delete old indices
GET _cat/snapshots/cs-automated?v&s=start_epoch:desc
GET _snapshot/cs-automated/2021-04-08t08-50-34.1efe36ab-ed7d-4dc7-b160-6014453d8774
DELETE qlog-fff-202104-1
DELETE qlog-fff-202105-000002Check the snapshot or not before deletion