Skip to content

Commit 5e81fa0

Browse files
srnagaribrandes
andauthored
V2 Encryption Bug 4MB file downloads (#41813) (#42213)
* V2 Encryption Bug 4MB file downloads (#41813) * testing, need to get this branch on my other workstation * adding broken test * updating tests * tests with logging * updating test and adding fix * making test paramaterized and adding change for only encryption v2 downloads * fixing imports * wip context addition * context implementation and sync / async tests * adding back unpulled tests * consolidating one of the async util methods * addressing all comments besides 1 * removing leftover changes and adding back test for V2 * kyle test * adjusting unencrypted blob length calculation * recording tests and fixing style * addressing comments * changelog update * fixing messed up recorded test * adding more test cases and updating adjustment method signature * adding missed recordings and fixing style * changing offsetAdjustment to a long to prevent int overflow, and added support for unencrypted blob length for blobinputstream * adding test for integer overflow, removing redundant test, adding helper method to blobinputstream * adjusting changelong * removing unused imports * changing blobinputstream util method to use long instead of Long * update versions * update readmes * update changelogs --------- Co-authored-by: Isabelle <141270045+ibrandes@users.noreply.github.com>
1 parent 5076a39 commit 5e81fa0

File tree

31 files changed

+333
-57
lines changed

31 files changed

+333
-57
lines changed

eng/versioning/version_client.txt

+3-3
Original file line numberDiff line numberDiff line change
@@ -186,12 +186,12 @@ com.azure:azure-security-keyvault-perf;1.0.0-beta.1;1.0.0-beta.1
186186
com.azure:azure-sdk-template;1.1.1234;1.2.2-beta.1
187187
com.azure:azure-sdk-template-two;1.0.0-beta.1;1.0.0-beta.1
188188
com.azure:azure-sdk-template-three;1.0.0-beta.1;1.0.0-beta.1
189-
com.azure:azure-storage-blob;12.27.1;12.28.0
189+
com.azure:azure-storage-blob;12.27.1;12.28.1
190190
com.azure:azure-storage-blob-batch;12.23.1;12.24.0
191191
com.azure:azure-storage-blob-changefeed;12.0.0-beta.24;12.0.0-beta.25
192-
com.azure:azure-storage-blob-cryptography;12.26.1;12.27.0
192+
com.azure:azure-storage-blob-cryptography;12.26.1;12.27.1
193193
com.azure:azure-storage-blob-nio;12.0.0-beta.25;12.0.0-beta.26
194-
com.azure:azure-storage-common;12.26.1;12.27.0
194+
com.azure:azure-storage-common;12.26.1;12.27.1
195195
com.azure:azure-storage-file-share;12.23.1;12.24.0
196196
com.azure:azure-storage-file-datalake;12.20.1;12.21.0
197197
com.azure:azure-storage-internal-avro;12.12.1;12.13.0

sdk/storage/azure-storage-blob-batch/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@
6565
<dependency>
6666
<groupId>com.azure</groupId>
6767
<artifactId>azure-storage-blob</artifactId>
68-
<version>12.28.0</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
68+
<version>12.28.1</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
6969
</dependency>
7070

7171
<!-- Added this dependency to include necessary annotations used by reactor core.
@@ -81,7 +81,7 @@
8181
<dependency>
8282
<groupId>com.azure</groupId>
8383
<artifactId>azure-storage-common</artifactId>
84-
<version>12.27.0</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
84+
<version>12.27.1</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
8585
<classifier>tests</classifier>
8686
<type>test-jar</type>
8787
<scope>test</scope>

sdk/storage/azure-storage-blob-changefeed/pom.xml

+2-2
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@
7777
<dependency>
7878
<groupId>com.azure</groupId>
7979
<artifactId>azure-storage-blob</artifactId>
80-
<version>12.28.0</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
80+
<version>12.28.1</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
8181
</dependency>
8282

8383
<!-- Added this dependency to include necessary annotations used by reactor core.
@@ -93,7 +93,7 @@
9393
<dependency>
9494
<groupId>com.azure</groupId>
9595
<artifactId>azure-storage-common</artifactId>
96-
<version>12.27.0</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
96+
<version>12.27.1</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
9797
<classifier>tests</classifier>
9898
<type>test-jar</type>
9999
<scope>test</scope>

sdk/storage/azure-storage-blob-cryptography/CHANGELOG.md

+9
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
11
# Release History
22

3+
## 12.27.1 (2024-10-08)
4+
5+
### Bugs Fixed
6+
- Fixed a bug where downloadToFile and openInputStream was throwing an InvalidRange exception if the target file size was a multiple of the
7+
authenticated region length.
8+
9+
#### Dependency Updates
10+
- Upgraded `azure-storage-blob` from `12.28.0` to version `12.28.1`.
11+
312
## 12.27.0 (2024-09-17)
413

514
### Features Added

sdk/storage/azure-storage-blob-cryptography/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ add the direct dependency to your project as follows.
5858
<dependency>
5959
<groupId>com.azure</groupId>
6060
<artifactId>azure-storage-blob-cryptography</artifactId>
61-
<version>12.27.0</version>
61+
<version>12.27.1</version>
6262
</dependency>
6363
```
6464
[//]: # ({x-version-update-end})

sdk/storage/azure-storage-blob-cryptography/assets.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
"AssetsRepo": "Azure/azure-sdk-assets",
33
"AssetsRepoPrefixPath": "java",
44
"TagPrefix": "java/storage/azure-storage-blob-cryptography",
5-
"Tag": "java/storage/azure-storage-blob-cryptography_e245db55c0"
5+
"Tag": "java/storage/azure-storage-blob-cryptography_9fd41be58a"
66
}

sdk/storage/azure-storage-blob-cryptography/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
<groupId>com.azure</groupId>
1515
<artifactId>azure-storage-blob-cryptography</artifactId>
16-
<version>12.27.0</version> <!-- {x-version-update;com.azure:azure-storage-blob-cryptography;current} -->
16+
<version>12.27.1</version> <!-- {x-version-update;com.azure:azure-storage-blob-cryptography;current} -->
1717

1818
<name>Microsoft Azure client library for Blob Storage cryptography</name>
1919
<description>This module contains client library for Microsoft Azure Blob Storage cryptography.</description>
@@ -71,13 +71,13 @@
7171
<dependency>
7272
<groupId>com.azure</groupId>
7373
<artifactId>azure-storage-blob</artifactId>
74-
<version>12.28.0</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
74+
<version>12.28.1</version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
7575
</dependency>
7676

7777
<dependency>
7878
<groupId>com.azure</groupId>
7979
<artifactId>azure-storage-common</artifactId>
80-
<version>12.27.0</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
80+
<version>12.27.1</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
8181
<classifier>tests</classifier>
8282
<type>test-jar</type>
8383
<scope>test</scope>

sdk/storage/azure-storage-blob-cryptography/src/main/java/com/azure/storage/blob/specialized/cryptography/EncryptedBlobAsyncClient.java

+7-2
Original file line numberDiff line numberDiff line change
@@ -750,9 +750,14 @@ private <T> Mono<T> populateRequestConditionsAndContext(BlobRequestConditions re
750750

751751
String encryptionDataKey = StorageImplUtils.getEncryptionDataKey(response.getValue().getMetadata());
752752
if (encryptionDataKey != null) {
753-
result = result.contextWrite(context -> context.put(ENCRYPTION_DATA_KEY,
754-
EncryptionData.getAndValidateEncryptionData(encryptionDataKey, requiresEncryption)));
753+
EncryptionData encryptionData = EncryptionData.getAndValidateEncryptionData(encryptionDataKey,
754+
requiresEncryption);
755+
756+
result = result.contextWrite(context -> context.put(ENCRYPTION_DATA_KEY, encryptionData))
757+
.contextWrite(context -> context.put(Constants.ADJUSTED_BLOB_LENGTH_KEY,
758+
EncryptedBlobLength.computeAdjustedBlobLength(encryptionData, response.getValue().getBlobSize())));
755759
}
760+
756761
return result;
757762
});
758763
}

sdk/storage/azure-storage-blob-cryptography/src/main/java/com/azure/storage/blob/specialized/cryptography/EncryptedBlobClient.java

+6-3
Original file line numberDiff line numberDiff line change
@@ -489,8 +489,12 @@ private Context populateRequestConditionsAndContext(BlobRequestConditions reques
489489

490490
String encryptionDataKey = StorageImplUtils.getEncryptionDataKey(initialProperties.getMetadata());
491491
if (encryptionDataKey != null) {
492-
context = context.addData(ENCRYPTION_DATA_KEY, EncryptionData.getAndValidateEncryptionData(
493-
encryptionDataKey, encryptedBlobAsyncClient.isEncryptionRequired()));
492+
EncryptionData encryptionData = EncryptionData.getAndValidateEncryptionData(
493+
encryptionDataKey, encryptedBlobAsyncClient.isEncryptionRequired());
494+
495+
context = context.addData(ENCRYPTION_DATA_KEY, encryptionData)
496+
.addData(Constants.ADJUSTED_BLOB_LENGTH_KEY,
497+
EncryptedBlobLength.computeAdjustedBlobLength(encryptionData, initialProperties.getBlobSize()));
494498
}
495499
return context;
496500
}
@@ -572,5 +576,4 @@ public BlobQueryResponse queryWithResponse(BlobQueryOptions queryOptions,
572576
BlobClientSideEncryptionOptions getClientSideEncryptionOptions() {
573577
return encryptedBlobAsyncClient.getClientSideEncryptionOptions();
574578
}
575-
576579
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT License.
3+
4+
package com.azure.storage.blob.specialized.cryptography;
5+
6+
import com.azure.core.util.logging.ClientLogger;
7+
8+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.ENCRYPTION_PROTOCOL_V1;
9+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.ENCRYPTION_PROTOCOL_V2;
10+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.ENCRYPTION_PROTOCOL_V2_1;
11+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.NONCE_LENGTH;
12+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.TAG_LENGTH;
13+
14+
/**
15+
* This class provides helper methods for adjusting encrypted downloads.
16+
*/
17+
final class EncryptedBlobLength {
18+
private static final ClientLogger LOGGER = new ClientLogger(EncryptedBlobLength.class);
19+
20+
static Long computeAdjustedBlobLength(EncryptionData encryptionData, Long encryptedLength) {
21+
switch (encryptionData.getEncryptionAgent().getProtocol()) {
22+
/*
23+
Technically, the total unencrypted length may be different for v1,
24+
but because this helper method is only used for partitioning ranged downloads,
25+
the size does not need to be adjusted for v1.
26+
*/
27+
case ENCRYPTION_PROTOCOL_V1:
28+
return encryptedLength;
29+
case ENCRYPTION_PROTOCOL_V2:
30+
case ENCRYPTION_PROTOCOL_V2_1:
31+
long regionLength = encryptionData.getEncryptedRegionInfo().getDataLength();
32+
long region = (long) Math.ceil((double) encryptedLength / (double) (regionLength + NONCE_LENGTH + TAG_LENGTH));
33+
long offset = (NONCE_LENGTH + TAG_LENGTH) * region;
34+
return encryptedLength - offset;
35+
default:
36+
throw LOGGER.logExceptionAsError(new IllegalArgumentException("Unexpected protocol version"));
37+
}
38+
}
39+
}

sdk/storage/azure-storage-blob-cryptography/src/main/java/com/azure/storage/blob/specialized/cryptography/EncryptedBlobRange.java

+5-5
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ final class EncryptedBlobRange {
3535
* 0-31 for v1
3636
* 0-(4mb+12-1)
3737
*/
38-
private final int offsetAdjustment;
38+
private final Long offsetAdjustment;
3939

4040
/**
4141
* How many bytes to download, including the adjustments for encryption block boundaries and the IV.
@@ -78,7 +78,7 @@ static EncryptedBlobRange getEncryptedBlobRangeFromHeader(String stringRange, En
7878
EncryptedBlobRange(BlobRange originalRange, EncryptionData encryptionData) {
7979
if (originalRange == null) {
8080
this.originalRange = new BlobRange(0);
81-
this.offsetAdjustment = 0;
81+
this.offsetAdjustment = 0L;
8282
this.amountPlaintextToSkip = 0; // In cases where this block is executed, this value does not matter
8383
return;
8484
}
@@ -111,7 +111,7 @@ static EncryptedBlobRange getEncryptedBlobRangeFromHeader(String stringRange, En
111111
}
112112
}
113113

114-
this.offsetAdjustment = tempOffsetAdjustment;
114+
this.offsetAdjustment = (long) tempOffsetAdjustment;
115115

116116
/*
117117
Align adjustedDownloadCount with encryption block boundary at the end of the range. Note that it is impossible
@@ -153,7 +153,7 @@ static EncryptedBlobRange getEncryptedBlobRangeFromHeader(String stringRange, En
153153
}
154154

155155
// Offset adjustment is difference in two starting values
156-
this.offsetAdjustment = (int) (originalRange.getOffset() - regionStartOffset);
156+
this.offsetAdjustment = (originalRange.getOffset() - regionStartOffset);
157157

158158
break;
159159
default:
@@ -171,7 +171,7 @@ BlobRange getOriginalRange() {
171171
/**
172172
* @return Offset from beginning of BlobRange, 0-31.
173173
*/
174-
int getOffsetAdjustment() {
174+
Long getOffsetAdjustment() {
175175
return this.offsetAdjustment;
176176
}
177177

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved.
2+
// Licensed under the MIT License.
3+
4+
package com.azure.storage.blob.specialized.cryptography;
5+
6+
import com.azure.storage.common.implementation.Constants;
7+
import org.junit.jupiter.api.Test;
8+
import org.junit.jupiter.params.ParameterizedTest;
9+
import org.junit.jupiter.params.provider.Arguments;
10+
import org.junit.jupiter.params.provider.MethodSource;
11+
12+
import java.util.stream.Stream;
13+
14+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.GCM_ENCRYPTION_REGION_LENGTH;
15+
import static com.azure.storage.blob.specialized.cryptography.CryptographyConstants.NONCE_LENGTH;
16+
import static org.junit.jupiter.api.Assertions.assertEquals;
17+
import static org.junit.jupiter.api.Assertions.assertThrows;
18+
19+
public class EncryptedBlobLengthTests extends BlobCryptographyTestBase {
20+
21+
//to prevent having to cast int -> long everywhere
22+
private static final long FOUR_MB = 4 * Constants.MB;
23+
private static final long SIXTEEN_MB = 16 * Constants.MB;
24+
private static final long ONE_MB = Constants.MB;
25+
private static final long ONE_KB = Constants.KB;
26+
27+
@ParameterizedTest
28+
@MethodSource("correctAdjustedLengthV1Supplier")
29+
public void correctAdjustedLengthV1(Long encryptedLength) {
30+
EncryptionData encryptionData = new EncryptionData();
31+
encryptionData.setEncryptionAgent(new EncryptionAgent("1.0", null));
32+
33+
Long newLength = EncryptedBlobLength.computeAdjustedBlobLength(encryptionData, encryptedLength);
34+
assertEquals(encryptedLength, newLength);
35+
}
36+
37+
private static Stream<Arguments> correctAdjustedLengthV1Supplier() {
38+
return Stream.of(
39+
Arguments.of(FOUR_MB),
40+
Arguments.of(SIXTEEN_MB)
41+
);
42+
}
43+
44+
@ParameterizedTest
45+
@MethodSource("correctAdjustedLengthV2Supplier")
46+
public void correctAdjustedLengthV2(Long encryptedLength, Long expectedLength) {
47+
EncryptionData encryptionData = new EncryptionData();
48+
encryptionData.setEncryptionAgent(new EncryptionAgent("2.0", null));
49+
encryptionData.setEncryptedRegionInfo(new EncryptedRegionInfo(GCM_ENCRYPTION_REGION_LENGTH, NONCE_LENGTH));
50+
51+
Long newLength = EncryptedBlobLength.computeAdjustedBlobLength(encryptionData, encryptedLength);
52+
assertEquals(expectedLength, newLength);
53+
}
54+
55+
private static Stream<Arguments> correctAdjustedLengthV2Supplier() {
56+
return Stream.of(
57+
Arguments.of(FOUR_MB + 28, FOUR_MB),
58+
Arguments.of(FOUR_MB + 57, FOUR_MB + 1),
59+
Arguments.of(SIXTEEN_MB + 112, SIXTEEN_MB),
60+
Arguments.of(28L, 0L),
61+
Arguments.of(0L, 0L),
62+
Arguments.of(ONE_MB + 28, ONE_MB)
63+
);
64+
}
65+
66+
@ParameterizedTest
67+
@MethodSource("correctAdjustedLengthVariableRegionSupplier")
68+
public void correctAdjustedLengthVariableRegion(Long encryptedLength, Long expectedLength, Long regionLength) {
69+
EncryptionData encryptionData = new EncryptionData();
70+
encryptionData.setEncryptionAgent(new EncryptionAgent("2.1", null));
71+
encryptionData.setEncryptedRegionInfo(new EncryptedRegionInfo(regionLength, NONCE_LENGTH));
72+
73+
Long newLength = EncryptedBlobLength.computeAdjustedBlobLength(encryptionData, encryptedLength);
74+
assertEquals(expectedLength, newLength);
75+
}
76+
77+
private static Stream<Arguments> correctAdjustedLengthVariableRegionSupplier() {
78+
return Stream.of(
79+
Arguments.of(FOUR_MB + 112, FOUR_MB, ONE_MB),
80+
Arguments.of(SIXTEEN_MB + 448, SIXTEEN_MB, ONE_MB),
81+
Arguments.of(FOUR_MB + 114688, FOUR_MB, ONE_KB),
82+
Arguments.of(SIXTEEN_MB + 458752, SIXTEEN_MB, ONE_KB),
83+
Arguments.of(FOUR_MB + 448, FOUR_MB, 256 * ONE_KB),
84+
Arguments.of(SIXTEEN_MB + 1792, SIXTEEN_MB, 256 * ONE_KB),
85+
Arguments.of(ONE_MB + 28672, ONE_MB, ONE_KB),
86+
Arguments.of(ONE_MB + 28, ONE_MB, ONE_MB)
87+
);
88+
}
89+
90+
@Test
91+
public void badProtocol() {
92+
EncryptionData encryptionData = new EncryptionData();
93+
encryptionData.setEncryptionAgent(new EncryptionAgent("garbage", null));
94+
95+
assertThrows(IllegalArgumentException.class, () -> EncryptedBlobLength.computeAdjustedBlobLength(encryptionData,
96+
null));
97+
}
98+
99+
}

sdk/storage/azure-storage-blob-cryptography/src/test/java/com/azure/storage/blob/specialized/cryptography/EncryptedBlobRangeTests.java

+6-6
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,7 @@ public void encryptedBlobRangeFromEncryptionData(int regionLength) {
114114

115115
@ParameterizedTest
116116
@MethodSource("provideRanges")
117-
public void testAdjustedBlobRange(int originalOffset, long originalCount, int expectedNewOffset, long expectedNewCount) {
118-
int regionLength = 16;
117+
public void testAdjustedBlobRange(long originalOffset, long originalCount, long expectedNewOffset, long expectedNewCount, int regionLength) {
119118
EncryptionData encryptionData = new EncryptionData()
120119
.setEncryptionAgent(new EncryptionAgent(ENCRYPTION_PROTOCOL_V2, EncryptionAlgorithm.AES_GCM_256))
121120
.setEncryptedRegionInfo(new EncryptedRegionInfo(regionLength, NONCE_LENGTH));
@@ -131,10 +130,11 @@ public void testAdjustedBlobRange(int originalOffset, long originalCount, int ex
131130

132131
private static Stream<Arguments> provideRanges() {
133132
return Stream.of(
134-
Arguments.of(5, 10, 0, 44), // Entirely within a single region, adjustment includes nonce and tag
135-
Arguments.of(16, 16, 44, 44), // Exactly one region
136-
Arguments.of(15, 35, 0, 176), // Straddles across four regions
137-
Arguments.of(32, 15, 88, 44) // Starts exactly at the second region
133+
Arguments.of(5L, 10L, 0L, 44L, 16), // Entirely within a single region, adjustment includes nonce and tag
134+
Arguments.of(16L, 16L, 44L, 44L, 16), // Exactly one region
135+
Arguments.of(15L, 35L, 0L, 176L, 16), // Straddles across four regions
136+
Arguments.of(32L, 15L, 88L, 44L, 16), // Starts exactly at the second region
137+
Arguments.of(3674210304L, 10066367L, 8818104720L, 24159312L, 20) // Tests for integer overflow with a very large blob
138138
);
139139
}
140140
}

0 commit comments

Comments
 (0)