top of page


  • Philip Osadebay - Tech Journalist

Unveiling GPTBot: OpenAI's innovative web crawler for automated data extraction across the internet

In the age of boundless information, OpenAI is seen as a transformative tool to reshape data collection ethics. OpenAI has addressed concerns about privacy and intellectual property from data collection on public websites by introducing a new web tool named GPTBot.

OpenAI acknowledges the importance of offering options to website administrators regarding GPTBot's access to their platforms. Providing access is viewed as a collaboration to enhance the accuracy of AI models, thus strengthening their capabilities and reinforcing security measures.

To promote transparency, OpenAI has released the IP address range associated with GPTBot's activities. This disclosure facilitates the identification of the bot's actions but also allows for blocking its access if deemed necessary.

These initiatives of OpenAI's is a response to criticism directed at AI model operators who have been accused of collecting data without explicit consent. It's been argued that such practices within the industry might infringe upon intellectual rights. There have been a demand for AI entities to implement more comprehensive mechanisms, granting website owners and data custodians the ability to decide whether their content can be used.

Kickstarter, a crowdfunding platform, recently introduced regulations concerning AI projects. One significant requirement of the crowdfunding platform is that projects utilising external data sources must present evidence of valid licensing agreements and consent obtained from the source websites. Failure to meet this obligation renders projects ineligible for listing on Kickstarter.

OpenAI is aimed at a major transformation, characterised by migrating the foundational ChatGPT layer to GPT-4. Additionally, enhancements to the Code Interpreter plugin will encompass support for uploading multiple files to prompts, reflecting OpenAI's steadfast commitment to continuous enhancement and innovation.


bottom of page