In this era of rapid technological advancement, businesses are faced with a new and complex question: Should they grant access to web crawlers employed by conversational artificial intelligence (AI) technologies like ChatGPT? The decision to allow these AI-powered chatbots access to their websites presents a conundrum for companies, with implications for their intellectual property and online visibility.
The dilemma – intellectual property vs. exposure
Imagine running a successful business that heavily relies on online referrals from traditional search engines. Your business model is suddenly disrupted by AI-powered tools that scour the web for answers. Allowing these AI chatbots access to your website can ensure that your company is mentioned in their responses to user queries, potentially driving more traffic and revenue. However, it also means sharing your intellectual property with these large-language models (LLMs).
On the other hand, if you restrict access to your website, you protect your intellectual property but risk being excluded from the chatbots’ answers. So, which option should businesses choose: embrace AI web crawlers or shut them out?
The rise of chatGPT and competing technologies
This dilemma has become a reality for businesses worldwide, especially with the sensational introduction of ChatGPT in November 2022. ChatGPT, powered by OpenAI, has not only transformed information discovery but also influenced major players like Microsoft, which integrated ChatGPT into its conversational search engine Bing. In response, Google is planning to incorporate generative AI into its search results through its own chatbot, Bard.
While some companies remain hesitant to grant access to AI-powered web crawlers, others are realizing the benefits of embracing these chatbots.
Capitalizing on ChatGPT citations
Companies like Logikcull and BlendJet have embraced AI chatbots and reaped the rewards. Logikcull, a specialist in legal technology, detected its first referral from ChatGPT in March, and this opportunity has since become a significant revenue driver, with about 5% of referrals mentioning ChatGPT citations. Andy Wilson, CEO of Logikcull, sees this channel driving $100,000 in subscription revenue each month.
BlendJet, a portable blender company, also experienced an increase in website traffic after being cited in ChatGPT’s answers to various queries. These success stories highlight the potential benefits of cooperating with AI web crawlers.
The quirk in ChatGPT’s responses
Some companies have benefited from a unique aspect of ChatGPT’s responses. ChatGPT’s knowledge base is based on a pre-2021 web crawl, meaning it relies on outdated data. Companies like BlendJet are optimizing their websites to improve their chances of being cited in the chatbot’s updated knowledge base.
Who should grant access to AI web crawlers?
The decision to allow LLM-powered chatbots access to your website should depend on your business’s nature and how much it stands to gain from being included in their responses.
Lily Ray, an expert in search engine optimization (SEO) and senior director at marketing agency Amsive advises that the choice should align with how your company generates revenue. If your revenue comes from online content, granting access to AI chatbots might not be advantageous. However, for companies aiming to enhance their online branding, allowing access to chatbot web crawlers makes sense.
Optimizing your site for AI searches
For businesses considering opening their doors to AI chatbot web crawlers, optimizing their websites is crucial. This process is similar to traditional SEO work and involves focusing on brand- and entity-oriented SEO strategies.
Lily Ray emphasizes the importance of serving information that is easily linked to questions, utilizing keywords judiciously, and presenting content in a concise and conversational manner. This approach aligns with how LLMs operate, predicting the next likely word based on context.
Monitoring and embracing the future
If your business decides to share intellectual property with AI chatbots for increased visibility, monitoring website traffic to detect web crawlers is essential. For example, BlendJet’s CEO, Ryan Pamplin, identified OpenAI’s web crawler by its specific IP address and interaction style, allowing it access. Businesses are recognizing the value of these new AI-driven search tools and are adapting to this evolving landscape.
The decision of whether to grant AI chatbot web crawlers access to your website hinges on your business model and goals. While there are potential risks in sharing intellectual property, the rewards in terms of increased visibility and revenue can be substantial. As AI continues to shape the future of online interactions, businesses must carefully weigh their options and adapt their online strategies to thrive in this new landscape.
