What makes up a search engine?
In general, web crawlers are also called bots or spiders. They are programmed to index the content of websites across the internet to validate and show the most relevant and reliable search results.
When you search for something general such as, a company name on LinkedIn or a person on Twitter, ensure you are using a notion of popularity on the results: a result with more followers has more chance to be what the users are searching for. Solving this problem has been the primary focus at Keyspider. We address this by developing a configurable, yet easy to understand ranking algorithm where developers have complete control on the search configuration. You can configure the popularity via Custom Ranking, make sure you can address a complex problem like typosquatting, promote featured items and so on.
There is an endless way to configure the ranking, depending on the use case.
Collection and Views
Collection: A collection is where all the data/information of the web is stored. It is the data repository that holds all the data of the source ready for crawling.
Views: Views can help you optimize the search results by creating search interfaces between the user and the search engine.
Create a Search Engine
As organizations handle multiple products/services, Keyspider enables you to create multiple search engines for better performance and efficient site management.
For example, if you own an eCommerce site and a content site, you can create two search engines for each site or a single search engine for both the websites and manage them effortlessly in your Keyspider dashboard.
Keyspider uses the Crawler-based Engine within the search. Keyspider customizes the data sources in the Engine.
Procedure to Create a Search Engine
At Keyspider, Engine is the top-level parent, followed by Collection and View.
Inside the engine, you can add data sources, customize the data source and You can create multiple Engines.
In the following screen, the details are as follows:
- Community is the Engine
- community_collection is the Collection and
- default_view is the View
To create an Engine, follow the steps given below:
- Go to Account Settings as shown below:
Click Engines.
Add Engine
Click Add Engine.
Enter the Collection name as shown below:
Every Engine has one Collection and View, without which you cannot proceed.
Select the Data source type as ‘Website’
Enter the Website URL. For demo purpose, give the Website URL as http://demo.com.
This is shown below:
Note: By default the ‘Start Indexing’ is enabled. The website indexing starts indexing immediately before going into the website. If you want customizations or configurations, you can disable ‘Start Indexing’.
Note: Check the option ‘Include sitemap’, to crawl only the sitemap URL and not any other thing.
Click Save.