Search queries

The e-commerce spider template supports a spider argument, search_queries, that allows you to define a different search query per line, and turns the input URLs into search requests for those queries.

For example, given the following input URLs:

https://a.example
https://b.example

And the following list of search queries:

foo bar
baz

By default, the spider would send 2 initial requests to those 2 input URLs, to try and find out how to build a search request for them, and if it succeeds, it will then send 4 search requests, 1 per combination of input URL and search query. For example:

https://a.example/search?q=foo+bar
https://a.example/search?q=baz
https://b.example/s/foo%20bar
https://b.example/s/baz

The default implementation uses a combination of HTML metadata, AI-based HTML form inspection and heuristics to find the most likely way to build a search request for a given website.

If this default implementation does not work as expected on a given website, you can write a page object to fix that.