Alright. So, welcome back everybody for another deep dive.
Yes.
Um, today we're going to be looking at how websites,
okay,
can allow themselves to be crawled by chat GPT.
Interesting.
And some of the benefits and really what this means for the future of search.
Um, and we're going to be looking at an article called Boost Visibility with Chat GPT. Allow your site to be crawled from aifungy.com
which was published on November 9th, 2024.
Gotcha.
And You know, one of the interesting points here is that there's actually different crawlers for chat GPT.
Oh.
Each with a specific job.
Wow. Really?
Yeah. And I I think a lot of people, you know, when they think of chat GPT, they think of it as like,
you know, the chatbot that you can use to have a conversation with or to write,
you know, poems and things like that.
Right. Right.
But increasingly, ChatGpt is becoming a source for information
and a go-to source when people need to find something out online.
It's it's really becoming a search engine in its own right.
Yeah. And so I guess the first question is why is this a big deal? Like why should people care if chat GPT is crawling their website?
It's all about visibility.
Okay,
if you want your content to be seen by the growing number of people who are using chat GPT,
right?
You need to make sure that chat GPT can actually find and understand your content.
So it's kind of like the early days of Google where you had to optimize your website to get ranked.
Yeah, you're ahead of the curve here.
Yeah. So how how does this work? Like what do you mean by crawling a website? Well, ChatGpt uses these things called crawlers, which are basically like little digital explorers that go out and visit websites, okay? And they read the content and they try to understand what the website is about. Okay.
And then they use that information to answer user queries and to train Chat GPT's AI model.
So, they're like little librarians going out Yeah.
gathering information.
That's a great analogy.
And bringing it back to ChatGpt.
Precisely.
So, these crawlers like how do they know where to go and what to do.
Well, that's where this thing called the robots.txt file comes in.
Okay.
Have you heard of this before?
I have heard of it.
Okay. So, this is basically like a set of instructions
for the crawlers. It tells them which parts of your website they're allowed to visit and which parts they should stay away from.
Got it.
It's like a gatekeeper for your website.
So, this is how you can say like, "Hey, Chad GPT, you're allowed to look at this page."
Exactly.
But don't look at this other page.
Precisely. And that's where things start to get really interesting.
Okay. So, before we get into how actually modify this robots.txt file.
Yeah,
I think it's worth talking about the different types of crawlers that chat GPT uses.
Okay.
Because as the article mentions a few specific ones,
right?
Um and they each seem to have a slightly different purpose.
Yeah, they do. So, you've got the OI search bot, which is kind of like the general researcher. Okay.
Its job is to crawl the web and gather a wide range of information that can be used in Chat GPT's responses.
So, if I ask Chat GPT what the capital of France is, is right.
It might be pulling that information from the OEI search bot.
Exactly.
Okay, cool.
So, that's the general one, but then there's also the chat GPT user crawler, which is a little more specialized. Okay.
This one is focused on finding content that directly answers user queries.
So, it's looking for specific answers
within the chat GPT interface.
So, if I ask Chat GPT,
Yeah.
uh how to bake a cake. Yeah.
This is the one that's going to be looking for that specific recipe.
That's the one.
Okay. Wow. And so, there's two crawlers and the article also mentions a third one called GPTBOT.
That's right.
What does this one do?
So, this one's a little bit different.
Okay.
It's not directly involved in answering user queries.
Okay.
Instead, its primary role is to collect data that's used to train Chat GPT's AI model.
Interesting. So, it's not giving answers.
It's like the teacher.
It's gathering the information to make
Yeah. It's helping chat GPT become smarter.
Chat GPT smarter. Wow. Okay. So, we've got these three different crawlers.
Yeah.
Each with with a different purpose,
each with a specific task.
And they're all out there scouring the web for information.
Scouring the web
to help chat GPT become the ultimate source of knowledge and information.
That's the goal.
Wow.
Yeah.
Okay. So, I guess the next question is, how do we as website owners? Yes.
Control these crawlers?
Yes. How do we manage them? How do we
how do we tell them what we want them to do?
That's the million-dollar question, and that's what we're going to dive into next.
All right. So, stay tuned and we will be right back.
We'll be back after the break. And that is the robots.txt file,
right?
That's how we tell them what to do.
Yeah. So, this is how we tell those crawlers what they're allowed to see and what they're not allowed to see. Exactly.
Okay. So, let's say, for example,
um I'm totally fine with the OI search bot crawling my whole site. Okay.
Because I want chat GPT to be able to use my content to answer questions. But I'm not comfortable with GPTBOT.
Okay.
Using my content to train its AI model,
okay,
can actually do that?
You absolutely can.
Can I block specific crawlers?
Yes, you can use the robots.txt file to specify exactly which crawlers are allowed to access which parts of your site.
Okay, so this is really where the power comes in.
Exactly. You have a lot of control here.
So let's say I wanted to block GPTbot from accessing my whole website.
Okay,
what would that look like in the robots.txt file?
So you would open up your robots.txt file,
okay,
which is usually located in the root directory of your say, and you would add the following lines. Okay. User agent,
TPTBOT. Okay.
Disallow.
Okay. So, user agent that tells the file which crawler we're talking about.
Exactly.
And then disallow.
That means that GPTbot is not allowed to access any pages on your website.
So, that forward slash means everything.
Exactly. It's like putting up a big do not enter sign.
I like it. It's like
for GPTO.
Yeah. You're not allowed in here. Keep out.
Okay. So, we can use this to block specific crawlers.
Yes.
Or we can use it to allow specific crawlers.
You can do all sorts of things. You can even block or allow crawlers from specific sections of your website.
Oh wow. Okay. So, we're going to get into the specific back into the nitty-gritty. Do
that in just a minute. But before we do, I I want to kind of zoom out for a second to talk about the bigger picture here because
if Chad GPT really is becoming the next Google.
Yeah.
Like what does this mean for the future of SEO?
That's a great question and I think it's something that everybody who has a website is thinking about right now.
Yeah. Because right now we're so used to optimizing for Google Yeah, we're all about keywords and backlinks, right?
Trying to game the algorithm.
Well, but if chat GPT is using a completely different system,
right, it's a different ballgame.
How do we need to change our approach?
Well, I think the key is to focus on creating content that is not only relevant to your target audience, but also easily understandable by AI.
Okay. So, what does that mean?
So, it means using clear and concise language.
Okay.
Structuring your content in a logical way. and using headings and subheadings to break up your text.
So, it's almost like we have to think about writing for two different audiences now.
Yeah. You have to think about humans and you have to think about AI.
Wow. Okay. So, it's not just about keywords anymore.
Not anymore.
It's really about understanding.
So, AI
reads and interprets.
Understanding the intent behind the search.
Yeah. And making sure that our content is aligned with that.
Making sure that your content can answer those questions.
This is a huge shift.
It's a paradigm shift
in how we think about creating content.
Absolutely.
For the web.
It's exciting though.
It is exciting. It's kind of scary.
It is a little bit scary,
but it's also really cool.
It's the future.
Okay. So, I think now that we have a good understanding of the why,
why
and kind of the bigger picture, I think it's time to really dive into the how.
Okay, let's do it.
Let's talk about
let's get specific
the specifics of how to modify that robots.txt file.
Let's get our hands dirty.
Okay, so let's get into the nitty-gritty,
right?
How do we actually modify this robot? txt file.
Well, the first step is to find it.
Okay. And where would I find that?
So, it's usually located in the root directory of your website.
Root directory.
Yeah. And if you're not sure how to access that, your web hosting provider should be able to help you out.
Okay. So, I've found my robots.txt file.
I've opened it up.
Okay.
And it's just a text file.
It is. It's a plain text file.
So, I can edit it.
Yeah, you can edit it with any text editor. Okay. Like notepad or text edit.
Okay. Cool. So, what do I actually write in this file.
Okay, so the basic structure of the robots.txt file is pretty simple.
Okay,
each line consists of two parts.
Okay,
you have the user agent which identifies the specific crawler you're giving instructions to. And then you have the allow or disallow directive which tells the crawler whether it's allowed to access certain parts of your site.
Okay, so let's go back to that example, the OI search bot, right?
If I want to allow the O AI searchbot to crawl my entire website.
Uhhuh.
What would that look like in the robots.txt file?
Okay. So, it would look like this. Okay. User agent oi search box.
Okay. So, that forward slash afterallow.
That means the entire website
means it can access everything.
Exactly. It's like giving it a free pass.
Okay. So, what if I want to block all of the crawlers from a specific section of my website?
Okay. So, let's say you have a blog that's located in a folder called blog.
Okay.
You would add these lines to your robots.txt file.
User agent, right?
Disallow.blog.
Okay. So, the asterisk means all
all crawlers.
All crawlers.
Yep.
And then blog means they can't access anything in that folder.
Exactly.
Okay. And can I do this with like subfolders as well?
Yeah. Specific pages. You can get really granular with it.
Wow. This is really cool.
It's powerful stuff.
It's like I have so much control.
You do
over what chat GPT can see and can't see.
You're the gatekeeper.
I like it.
You have the keys to the kingdom.
Well, this has been an incredibly insightful
It has
deep dive.
Yeah, I think we covered a lot of ground today.
Yeah, we went from,
you know, not really knowing anything about
robots.txt files and all these different crawlers, right?
To really understanding how we can leverage this knowledge
and the implications for the future
to kind of stay ahead of the curve.
Yeah. And how to optimize our websites.
Yeah.
For this new world of search.
So, for anyone listening out there who has a website. Yes, I highly encourage you check out the full article.
Go to aifundy.com
from aifungy.com.
Yeah. And read the full article.
Yeah. Boost visibility with chat GPT. Allow your site to be crawled.
It's a good one.
It's a great read. Um and thank you so much for joining us today.
Yeah, this was fun.
And we will see you all next time.
See you next time.