Skip to content

Commit d8668c5

Browse files
jmilehamsmudgesamandmoore
authored
Add URL wildcard feature (fixes #47) (#53)
* Add URL wildcard feature (fixes #47) * Replace confusing/inaccurate test * Add another illustrative test of basic auth behavior * Correctly describe test * collapse unused constant * Rename general purpose boundary chars away from wildcard * Simplify boundary chars * Fix bug with trailing slashes * extra newline * test double-star behavior * moar tests * Note breaking changes in post_install_message * Bump version to 2.0 * Clarify language * enshorten * Update lib/webvalve/version.rb Co-authored-by: Nathan Griffith <nathan@betterment.com> * Better section title * Switch to Addressable::Template strategy * Factor out `#template` for legibility * Fix URL edge case and add tests * Typo in code comments * more succinct * also more succinct * Update README to match Addressable::Template capabilities * Better error message * Better comment * Update spec/webvalve/service_url_converter_spec.rb * Match multiple query params! * Revert "Match multiple query params!" This reverts commit 878fea0. * Revert "Better comment" This reverts commit d1f8cb5. * Revert "Better error message" This reverts commit 6ed0170. * Revert "also more succinct" This reverts commit e20c482. * Revert "more succinct" This reverts commit 96c0aed. * Revert "Typo in code comments" This reverts commit cb35fe3. * Revert "Factor out `#template` for legibility" This reverts commit ad2775c. * Revert "Switch to Addressable::Template strategy" This reverts commit 2deb420. * Don't rely on Addressable for tests when it's not a dependency * 2.0.0 changelog entry --------- Co-authored-by: Nathan Griffith <nathan@betterment.com> Co-authored-by: Sam Moore <sam@betterment.com>
1 parent 97e8645 commit d8668c5

8 files changed

+297
-10
lines changed

CHANGELOG.md

+4
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ and this project aims to adhere to [Semantic Versioning](http://semver.org/spec/
99
### Added
1010
### Removed
1111

12+
## [2.0.0] - 2023-07-20
13+
### Added
14+
- Dynamic URL support via wildcards, Regexps, and Addressable::Templates
15+
1216
## [1.3.1] - 2023-07-20
1317
### Changed
1418
- Replace usage of deprecated `File.exists?` in generator

README.md

+37
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,43 @@ WebValve.register FakeBank, url: ENV.fetch("SOME_CUSTOM_API_URL")
218218
WebValve.register FakeBank, url: "https://some-service.com"
219219
```
220220

221+
## Dynamic URLs
222+
223+
If the service you are interacting with contains dynamic elements, e.g.
224+
an instance-specific subdomain, you can specify a wildcard in your url
225+
with the `*` character to match a series of zero or more characters
226+
within the same URL segment. For example:
227+
228+
```bash
229+
export BANK_API_URL=https://*.mybank.com/
230+
```
231+
232+
or
233+
234+
```ruby
235+
WebValve.register FakeBank, url: "https://*.mybank.com"
236+
```
237+
238+
Note: unlike filesystem globbing, `?` isn't respected to mean "exactly
239+
one character" because it's a URL delimiter character. Only `*` works
240+
for WebValve URL wildcards.
241+
242+
Alternatively you can use `Addressable::Template`s or `Regexp`s to
243+
specify dynamic URLs if they for some reason aren't a good fit for the
244+
wildcard syntax. Note that there is no `ENV` var support for these
245+
formats because there is no detection logic to determine a URL string is
246+
actually meant to represent a URL template or regexp. For example:
247+
248+
```ruby
249+
WebValve.register FakeBank, url: Addressable::Template.new("http://mybank.com{/path*}{?query}")
250+
```
251+
252+
or
253+
254+
```ruby
255+
WebValve.register FakeBank, url: %r{\Ahttp://mybank.com(/.*)?\z}
256+
```
257+
221258
## What's in a `FakeService`?
222259

223260
The definition of `FakeService` is really simple. It's just a

lib/webvalve/manager.rb

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
require 'webmock'
22
require 'singleton'
33
require 'set'
4+
require 'webvalve/service_url_converter'
45

56
module WebValve
67
ALWAYS_ENABLED_ENVS = %w(development test).freeze
@@ -148,7 +149,7 @@ def allowlist_service(config)
148149
end
149150

150151
def url_to_regexp(url)
151-
%r(\A#{Regexp.escape url})
152+
ServiceUrlConverter.new(url: url).regexp
152153
end
153154

154155
def ensure_non_duplicate_stub(config)

lib/webvalve/service_url_converter.rb

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
module WebValve
2+
class ServiceUrlConverter
3+
TOKEN_BOUNDARY_CHARS = Regexp.escape('.:/?#@&=').freeze
4+
WILDCARD_SUBSTITUTION = ('[^' + TOKEN_BOUNDARY_CHARS + ']*').freeze
5+
URL_PREFIX_BOUNDARY = ('[' + TOKEN_BOUNDARY_CHARS + ']').freeze
6+
URL_SUFFIX_PATTERN = ('((' + URL_PREFIX_BOUNDARY + '|(?<=' + URL_PREFIX_BOUNDARY + ')).*)?\z').freeze
7+
8+
attr_reader :url
9+
10+
def initialize(url:)
11+
@url = url
12+
end
13+
14+
def regexp
15+
if url.is_a?(String)
16+
regexp_string = Regexp.escape(url)
17+
substituted_regexp_string = regexp_string.gsub('\*', WILDCARD_SUBSTITUTION)
18+
%r(\A#{substituted_regexp_string}#{URL_SUFFIX_PATTERN})
19+
else
20+
url
21+
end
22+
end
23+
end
24+
end

lib/webvalve/version.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
module WebValve
2-
VERSION = "1.3.1"
2+
VERSION = "2.0.0"
33
end

spec/webvalve/manager_spec.rb

+18-8
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@
3838
end
3939

4040
describe '#setup' do
41+
let(:wildcard_substitution) { WebValve::ServiceUrlConverter::WILDCARD_SUBSTITUTION }
42+
let(:url_suffix_pattern) { WebValve::ServiceUrlConverter::URL_SUFFIX_PATTERN }
43+
4144
context 'when WebValve is disabled' do
4245
around do |ex|
4346
with_rails_env 'production' do
@@ -94,10 +97,17 @@
9497

9598
it 'allowlists configured urls in webmock' do
9699
allow(WebMock).to receive(:disable_net_connect!)
97-
results = [%r{\Ahttp://foo\.dev}, %r{\Ahttp://bar\.dev}]
100+
results = [
101+
%r{\Ahttp://foo\.dev#{url_suffix_pattern}},
102+
%r{\Ahttp://bar\.dev#{url_suffix_pattern}},
103+
%r{\Ahttp://bar\.#{wildcard_substitution}\.dev#{url_suffix_pattern}},
104+
%r{\Ahttp://bar\.dev/\?foo=bar#{url_suffix_pattern}}
105+
]
98106

99107
subject.allow_url 'http://foo.dev'
100108
subject.allow_url 'http://bar.dev'
109+
subject.allow_url 'http://bar.*.dev'
110+
subject.allow_url 'http://bar.dev/?foo=bar'
101111

102112
subject.setup
103113

@@ -115,7 +125,7 @@
115125
subject.setup
116126
end
117127

118-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
128+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
119129
expect(web_mock_stubble).to have_received(:to_rack)
120130
end
121131

@@ -153,7 +163,7 @@
153163
subject.register other_disabled_service.name
154164

155165
expect { subject.setup }.to_not raise_error
156-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev}).twice
166+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}}).twice
157167
end
158168
end
159169
end
@@ -208,8 +218,8 @@
208218
subject.setup
209219
end
210220

211-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
212-
expect(WebMock).not_to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev})
221+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
222+
expect(WebMock).not_to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev#{url_suffix_pattern}})
213223
expect(web_mock_stubble).to have_received(:to_rack).once
214224
end
215225

@@ -261,9 +271,9 @@
261271
subject.setup
262272
end
263273

264-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
265-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\-else\.dev})
266-
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev})
274+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
275+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\-else\.dev#{url_suffix_pattern}})
276+
expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev#{url_suffix_pattern}})
267277
expect(web_mock_stubble).to have_received(:to_rack).exactly(3).times
268278
end
269279
end
+194
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
require 'spec_helper'
2+
3+
RSpec.describe WebValve::ServiceUrlConverter do
4+
let(:url) { "http://bar.com" }
5+
6+
subject { described_class.new(url: url) }
7+
8+
describe '#regexp' do
9+
it "returns a regexp" do
10+
expect(subject.regexp).to be_a(Regexp)
11+
end
12+
13+
context "with a regexp" do
14+
let(:url) { %r{\Ahttp://foo\.com} }
15+
16+
it "returns the same object" do
17+
expect(subject.regexp).to be_a(Regexp)
18+
expect(subject.regexp).to equal(url)
19+
end
20+
end
21+
22+
context "with an empty url" do
23+
let(:url) { "" }
24+
25+
it "matches empty string" do
26+
expect("").to match(subject.regexp)
27+
end
28+
29+
it "matches a string starting with a URL delimiter because the rest is just interpreted as suffix" do
30+
expect(":do:do:dodo:do:do").to match(subject.regexp)
31+
end
32+
33+
it "doesn't match a string that doesn't start with a delimiter" do
34+
expect("jamietart:do:do:dodo:do:do").not_to match(subject.regexp)
35+
end
36+
end
37+
38+
context "with a boundary char on the end" do
39+
let(:url) { "http://bar.com/" }
40+
41+
it "matches arbitrary suffixes" do
42+
expect("http://bar.com/baz/bump/beep").to match(subject.regexp)
43+
end
44+
end
45+
46+
context "with multiple asterisks" do
47+
let(:url) { "http://bar.com/**/bump" }
48+
49+
it "matches like a single asterisk" do
50+
expect("http://bar.com/foo/bump").to match(subject.regexp)
51+
end
52+
53+
it "doesn't match like a filesystem glob" do
54+
expect("http://bar.com/foo/bar/bump").not_to match(subject.regexp)
55+
end
56+
end
57+
58+
context "with a trailing *" do
59+
let(:url) { "http://bar.com/*" }
60+
61+
it "matches when empty" do
62+
expect("http://bar.com/").to match(subject.regexp)
63+
end
64+
65+
it "matches when existing" do
66+
expect("http://bar.com/foobaloo").to match(subject.regexp)
67+
end
68+
69+
it "matches with additional tokens" do
70+
expect("http://bar.com/foobaloo/wink").to match(subject.regexp)
71+
end
72+
73+
it "doesn't match when missing the trailing slash tho" do
74+
expect("http://bar.com").not_to match(subject.regexp)
75+
end
76+
end
77+
78+
context "with a totally wildcarded protocol" do
79+
let(:url) { "*://bar.com" }
80+
81+
it "matches http" do
82+
expect("http://bar.com/").to match(subject.regexp)
83+
end
84+
85+
it "matches anything else" do
86+
expect("gopher://bar.com/").to match(subject.regexp)
87+
end
88+
89+
it "matches empty" do
90+
expect("://bar.com").to match(subject.regexp)
91+
end
92+
end
93+
94+
context "with a wildcarded partial protocol" do
95+
let(:url) { "http*://bar.com" }
96+
97+
it "matches empty" do
98+
expect("http://bar.com/").to match(subject.regexp)
99+
end
100+
101+
it "matches full" do
102+
expect("https://bar.com/").to match(subject.regexp)
103+
end
104+
end
105+
106+
context "with a TLD that is a substring of another TLD" do
107+
let(:url) { "http://bar.co" }
108+
109+
it "doesn't match a different TLD when extending" do
110+
expect("http://bar.com").not_to match(subject.regexp)
111+
end
112+
end
113+
114+
context "with a wildcard subdomain" do
115+
let(:url) { "http://*.bar.com" }
116+
117+
it "matches" do
118+
expect("http://foo.bar.com").to match(subject.regexp)
119+
end
120+
121+
it "doesn't match when too many subdomains" do
122+
expect("http://beep.foo.bar.com").not_to match(subject.regexp)
123+
end
124+
end
125+
126+
context "with a partial postfix wildcard subdomain" do
127+
let(:url) { "http://foo*.bar.com" }
128+
129+
it "matches when present" do
130+
expect("http://foobaz.bar.com").to match(subject.regexp)
131+
end
132+
133+
it "matches when empty" do
134+
expect("http://foo.bar.com").to match(subject.regexp)
135+
end
136+
137+
it "doesn't match when out of order" do
138+
expect("http://bazfoo.bar.com").not_to match(subject.regexp)
139+
end
140+
end
141+
142+
context "with a partial prefix wildcard subdomain" do
143+
let(:url) { "http://*baz.bar.com" }
144+
145+
it "matches when present" do
146+
expect("http://foobaz.bar.com").to match(subject.regexp)
147+
end
148+
149+
it "matches when empty" do
150+
expect("http://baz.bar.com").to match(subject.regexp)
151+
end
152+
end
153+
154+
context "with a wildcarded basic auth url" do
155+
let(:url) { "http://*:*@bar.com" }
156+
157+
it "matches when present" do
158+
expect("http://bilbo:baggins@bar.com").to match(subject.regexp)
159+
end
160+
161+
it "doesn't match when malformed" do
162+
expect("http://bilbobaggins@bar.com").not_to match(subject.regexp)
163+
end
164+
165+
it "doesn't match when missing password part" do
166+
expect("http://bilbo@bar.com").not_to match(subject.regexp)
167+
end
168+
end
169+
170+
context "with a wildcarded path" do
171+
let(:url) { "http://bar.com/*/whatever" }
172+
173+
it "matches with arbitrarily spicy but legal, non-URL-significant characters" do
174+
expect("http://bar.com/a0-_~[]!$'(),;%+/whatever").to match(subject.regexp)
175+
end
176+
177+
it "doesn't match when you throw a URL-significant char in there" do
178+
expect("http://bar.com/life=love/whatever").not_to match(subject.regexp)
179+
end
180+
end
181+
182+
context "with a wildcarded query param" do
183+
let(:url) { "http://bar.com/whatever?foo=*&bar=bump" }
184+
185+
it "matches when present" do
186+
expect("http://bar.com/whatever?foo=baz&bar=bump").to match(subject.regexp)
187+
end
188+
189+
it "doesn't match when you throw a URL-significant char in there" do
190+
expect("http://bar.com/whatever?foo=baz#&bar=bump").not_to match(subject.regexp)
191+
end
192+
end
193+
end
194+
end

webvalve.gemspec

+17
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,21 @@ Gem::Specification.new do |s|
3030
s.add_development_dependency "rails"
3131

3232
s.required_ruby_version = ">= 3.0.0"
33+
34+
s.post_install_message = <<~MSG
35+
Thanks for installing WebValve!
36+
37+
Note for upgraders: If you're upgrading from a version less than 2.0, service
38+
URL behavior has changed. Please verify that your app isn't relying on the
39+
previous behavior:
40+
41+
1. `*` characters are now interpreted as wildcards, enabling dynamic URL
42+
segments. In the unlikely event that your URLs use `*` literals, you'll need
43+
to URL encode them (`%2A`) both in your URL spec and at runtime.
44+
45+
2. URL suffix matching is now strict. For example, `BAR_URL=http://bar.co` will
46+
no longer match `https://bar.com`, but it will match `http://bar.co/foo`. If
47+
you need to preserve the previous behavior, you can add a trailing `*` to
48+
your URL spec, e.g. `BAR_URL=http://bar.co*`.
49+
MSG
3350
end

0 commit comments

Comments
 (0)