Skip to content

Commit 03273b5

Browse files
sachinpkaleSachin Kale
and
Sachin Kale
authored
[Backport 2.x] [Remote Store] Backport remote segment store changes (#4380)
* [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add index level setting for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (#3703) * Inject remote store in IndexShard instead of RemoteStoreRefreshListener Signed-off-by: Sachin Kale <kalsac@amazon.com> * Pass supplier of RepositoriesService to RemoteDirectoryFactory Signed-off-by: Sachin Kale <kalsac@amazon.com> * Create isRemoteStoreEnabled function for IndexShard Signed-off-by: Sachin Kale <kalsac@amazon.com> * Explicitly close remoteStore on indexShard close Signed-off-by: Sachin Kale <kalsac@amazon.com> * Change RemoteDirectory.close to a no-op Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add remote store restore API implementation (#3642) * Add remote restore API implementation Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add support to add nested settings for remote store (#4060) * Add support to add nested settings for remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add rest endpoint for remote store restore (#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add validator that forces segment replication type before enabling remote store (#4175) * Add validator that forces segment replication type before enabling remote store Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Change remote_store setting validation message to make it more clear (#4199) * Change remote_store setting validation message to make it more clear Signed-off-by: Sachin Kale <kalsac@amazon.com> * [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <kalsac@amazon.com> * Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240) * Use RemoteSegmentStoreDirectory instead of RemoteDirectory Signed-off-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>
1 parent ea5b946 commit 03273b5

File tree

58 files changed

+3627
-62
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+3627
-62
lines changed

CHANGELOG.md

+3-2
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
88
- Added @dreamer-89 as an Opensearch maintainer ([#4342](https://github.com/opensearch-project/OpenSearch/pull/4342))
99

1010
### Changed
11-
- Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
12-
- Update to Netty 4.1.80.Final ([#4359](https://github.com/opensearch-project/OpenSearch/pull/4359))
11+
- Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
12+
- Update to Netty 4.1.80.Final ([#4359](https://github.com/opensearch-project/OpenSearch/pull/4359))
13+
- Use RemoteSegmentStoreDirectory instead of RemoteDirectory ([#4240](https://github.com/opensearch-project/OpenSearch/pull/4240))
1314

1415
### Deprecated
1516

client/rest-high-level/src/test/java/org/opensearch/client/RestHighLevelClientTests.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -885,7 +885,8 @@ public void testApiNamingConventions() throws Exception {
885885
"nodes.hot_threads",
886886
"nodes.usage",
887887
"nodes.reload_secure_settings",
888-
"search_shards", };
888+
"search_shards",
889+
"remote_store.restore", };
889890
List<String> booleanReturnMethods = Arrays.asList("security.enable_user", "security.disable_user", "security.change_password");
890891
Set<String> deprecatedMethods = new HashSet<>();
891892
deprecatedMethods.add("indices.force_merge");
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"remote_store.restore":{
3+
"documentation":{
4+
"url": "https://opensearch.org/docs/latest/opensearch/rest-api/remote-store#restore",
5+
"description":"Restores from remote store."
6+
},
7+
"stability":"experimental",
8+
"url":{
9+
"paths":[
10+
{
11+
"path":"/_remotestore/_restore",
12+
"methods":[
13+
"POST"
14+
]
15+
}
16+
]
17+
},
18+
"params":{
19+
"cluster_manager_timeout":{
20+
"type":"time",
21+
"description":"Explicit operation timeout for connection to cluster-manager node"
22+
},
23+
"wait_for_completion":{
24+
"type":"boolean",
25+
"description":"Should this request wait until the operation has completed before returning",
26+
"default":false
27+
}
28+
},
29+
"body":{
30+
"description":"A comma separated list of index IDs",
31+
"required":true
32+
}
33+
}
34+
}

server/src/internalClusterTest/java/org/opensearch/index/shard/IndexShardIT.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -675,7 +675,8 @@ public static final IndexShard newIndexShard(
675675
() -> {},
676676
RetentionLeaseSyncer.EMPTY,
677677
cbs,
678-
SegmentReplicationCheckpointPublisher.EMPTY
678+
SegmentReplicationCheckpointPublisher.EMPTY,
679+
null
679680
);
680681
}
681682

server/src/main/java/org/opensearch/action/ActionModule.java

+12
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@
6161
import org.opensearch.action.admin.cluster.node.usage.TransportNodesUsageAction;
6262
import org.opensearch.action.admin.cluster.remote.RemoteInfoAction;
6363
import org.opensearch.action.admin.cluster.remote.TransportRemoteInfoAction;
64+
import org.opensearch.action.admin.cluster.remotestore.restore.RestoreRemoteStoreAction;
65+
import org.opensearch.action.admin.cluster.remotestore.restore.TransportRestoreRemoteStoreAction;
6466
import org.opensearch.action.admin.cluster.repositories.cleanup.CleanupRepositoryAction;
6567
import org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction;
6668
import org.opensearch.action.admin.cluster.repositories.delete.DeleteRepositoryAction;
@@ -261,6 +263,7 @@
261263
import org.opensearch.common.settings.IndexScopedSettings;
262264
import org.opensearch.common.settings.Settings;
263265
import org.opensearch.common.settings.SettingsFilter;
266+
import org.opensearch.common.util.FeatureFlags;
264267
import org.opensearch.index.seqno.RetentionLeaseActions;
265268
import org.opensearch.indices.SystemIndices;
266269
import org.opensearch.indices.breaker.CircuitBreakerService;
@@ -308,6 +311,7 @@
308311
import org.opensearch.rest.action.admin.cluster.RestPutStoredScriptAction;
309312
import org.opensearch.rest.action.admin.cluster.RestReloadSecureSettingsAction;
310313
import org.opensearch.rest.action.admin.cluster.RestRemoteClusterInfoAction;
314+
import org.opensearch.rest.action.admin.cluster.RestRestoreRemoteStoreAction;
311315
import org.opensearch.rest.action.admin.cluster.RestRestoreSnapshotAction;
312316
import org.opensearch.rest.action.admin.cluster.RestSnapshotsStatusAction;
313317
import org.opensearch.rest.action.admin.cluster.RestVerifyRepositoryAction;
@@ -657,6 +661,9 @@ public <Request extends ActionRequest, Response extends ActionResponse> void reg
657661
actions.register(DeleteDanglingIndexAction.INSTANCE, TransportDeleteDanglingIndexAction.class);
658662
actions.register(FindDanglingIndexAction.INSTANCE, TransportFindDanglingIndexAction.class);
659663

664+
// Remote Store
665+
actions.register(RestoreRemoteStoreAction.INSTANCE, TransportRestoreRemoteStoreAction.class);
666+
660667
return unmodifiableMap(actions.getRegistry());
661668
}
662669

@@ -842,6 +849,11 @@ public void initRestHandlers(Supplier<DiscoveryNodes> nodesInCluster) {
842849
}
843850
}
844851
registerHandler.accept(new RestCatAction(catActions));
852+
853+
// Remote Store APIs
854+
if (FeatureFlags.isEnabled(FeatureFlags.REMOTE_STORE)) {
855+
registerHandler.accept(new RestRestoreRemoteStoreAction());
856+
}
845857
}
846858

847859
@Override
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.action.admin.cluster.remotestore.restore;
10+
11+
import org.opensearch.action.ActionType;
12+
13+
/**
14+
* Restore remote store action
15+
*
16+
* @opensearch.internal
17+
*/
18+
public final class RestoreRemoteStoreAction extends ActionType<RestoreRemoteStoreResponse> {
19+
20+
public static final RestoreRemoteStoreAction INSTANCE = new RestoreRemoteStoreAction();
21+
public static final String NAME = "cluster:admin/remotestore/restore";
22+
23+
private RestoreRemoteStoreAction() {
24+
super(NAME, RestoreRemoteStoreResponse::new);
25+
}
26+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.action.admin.cluster.remotestore.restore;
10+
11+
import org.opensearch.action.ActionRequestValidationException;
12+
import org.opensearch.action.support.IndicesOptions;
13+
import org.opensearch.action.support.clustermanager.ClusterManagerNodeRequest;
14+
import org.opensearch.common.Strings;
15+
import org.opensearch.common.io.stream.StreamInput;
16+
import org.opensearch.common.io.stream.StreamOutput;
17+
import org.opensearch.common.xcontent.ToXContentObject;
18+
import org.opensearch.common.xcontent.XContentBuilder;
19+
20+
import java.io.IOException;
21+
import java.util.ArrayList;
22+
import java.util.Arrays;
23+
import java.util.List;
24+
import java.util.Map;
25+
import java.util.Objects;
26+
27+
import static org.opensearch.action.ValidateActions.addValidationError;
28+
29+
/**
30+
* Restore remote store request
31+
*
32+
* @opensearch.internal
33+
*/
34+
public class RestoreRemoteStoreRequest extends ClusterManagerNodeRequest<RestoreRemoteStoreRequest> implements ToXContentObject {
35+
36+
private String[] indices = Strings.EMPTY_ARRAY;
37+
private Boolean waitForCompletion;
38+
39+
public RestoreRemoteStoreRequest() {}
40+
41+
public RestoreRemoteStoreRequest(StreamInput in) throws IOException {
42+
super(in);
43+
indices = in.readStringArray();
44+
waitForCompletion = in.readOptionalBoolean();
45+
}
46+
47+
@Override
48+
public void writeTo(StreamOutput out) throws IOException {
49+
super.writeTo(out);
50+
out.writeStringArray(indices);
51+
out.writeOptionalBoolean(waitForCompletion);
52+
}
53+
54+
@Override
55+
public ActionRequestValidationException validate() {
56+
ActionRequestValidationException validationException = null;
57+
if (indices == null || indices.length == 0) {
58+
validationException = addValidationError("indices are missing", validationException);
59+
}
60+
return validationException;
61+
}
62+
63+
/**
64+
* Sets the list of indices that should be restored from the remote store
65+
* <p>
66+
* The list of indices supports multi-index syntax. For example: "+test*" ,"-test42" will index all indices with
67+
* prefix "test" except index "test42". Aliases are not supported. An empty list or {"_all"} will restore all open
68+
* indices in the cluster.
69+
*
70+
* @param indices list of indices
71+
* @return this request
72+
*/
73+
public RestoreRemoteStoreRequest indices(String... indices) {
74+
this.indices = indices;
75+
return this;
76+
}
77+
78+
/**
79+
* Sets the list of indices that should be restored from the remote store
80+
* <p>
81+
* The list of indices supports multi-index syntax. For example: "+test*" ,"-test42" will index all indices with
82+
* prefix "test" except index "test42". Aliases are not supported. An empty list or {"_all"} will restore all open
83+
* indices in the cluster.
84+
*
85+
* @param indices list of indices
86+
* @return this request
87+
*/
88+
public RestoreRemoteStoreRequest indices(List<String> indices) {
89+
this.indices = indices.toArray(new String[indices.size()]);
90+
return this;
91+
}
92+
93+
/**
94+
* Returns list of indices that should be restored from the remote store
95+
*/
96+
public String[] indices() {
97+
return indices;
98+
}
99+
100+
/**
101+
* If this parameter is set to true the operation will wait for completion of restore process before returning.
102+
*
103+
* @param waitForCompletion if true the operation will wait for completion
104+
* @return this request
105+
*/
106+
public RestoreRemoteStoreRequest waitForCompletion(boolean waitForCompletion) {
107+
this.waitForCompletion = waitForCompletion;
108+
return this;
109+
}
110+
111+
/**
112+
* Returns wait for completion setting
113+
*
114+
* @return true if the operation will wait for completion
115+
*/
116+
public boolean waitForCompletion() {
117+
return waitForCompletion;
118+
}
119+
120+
/**
121+
* Parses restore definition
122+
*
123+
* @param source restore definition
124+
* @return this request
125+
*/
126+
@SuppressWarnings("unchecked")
127+
public RestoreRemoteStoreRequest source(Map<String, Object> source) {
128+
for (Map.Entry<String, Object> entry : source.entrySet()) {
129+
String name = entry.getKey();
130+
if (name.equals("indices")) {
131+
if (entry.getValue() instanceof String) {
132+
indices(Strings.splitStringByCommaToArray((String) entry.getValue()));
133+
} else if (entry.getValue() instanceof ArrayList) {
134+
indices((ArrayList<String>) entry.getValue());
135+
} else {
136+
throw new IllegalArgumentException("malformed indices section, should be an array of strings");
137+
}
138+
} else {
139+
if (IndicesOptions.isIndicesOptions(name) == false) {
140+
throw new IllegalArgumentException("Unknown parameter " + name);
141+
}
142+
}
143+
}
144+
return this;
145+
}
146+
147+
@Override
148+
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
149+
builder.startObject();
150+
builder.startArray("indices");
151+
for (String index : indices) {
152+
builder.value(index);
153+
}
154+
builder.endArray();
155+
builder.endObject();
156+
return builder;
157+
}
158+
159+
@Override
160+
public String getDescription() {
161+
return "remote_store";
162+
}
163+
164+
@Override
165+
public boolean equals(Object o) {
166+
if (this == o) return true;
167+
if (o == null || getClass() != o.getClass()) return false;
168+
RestoreRemoteStoreRequest that = (RestoreRemoteStoreRequest) o;
169+
return waitForCompletion == that.waitForCompletion && Arrays.equals(indices, that.indices);
170+
}
171+
172+
@Override
173+
public int hashCode() {
174+
int result = Objects.hash(waitForCompletion);
175+
result = 31 * result + Arrays.hashCode(indices);
176+
return result;
177+
}
178+
179+
@Override
180+
public String toString() {
181+
return Strings.toString(this);
182+
}
183+
}

0 commit comments

Comments
 (0)