Services
Strategy
We find the clearest path from what your business needs to what your website should do.
What’s included
- Page strategy
- Landing page strategy
- Site architecture
- Content planning
- Wireframes
- Migration planning
Branding
We shape a distinct visual identity that works on screen first and carries through everything you make.
What’s included
- Visual identity
- Brand refresh
- Colour & typography
- Moodboards & direction
- Brand guidelines
- Digital brand systems
Recent work
Design
We turn the plan into a distinct, usable design that feels right for your brand and works across every screen.
What’s included
- Website design
- UI design
- Responsive design
- Design systems
- Visual direction
- Content presentation
Development
We build fast, reliable websites in Webflow or with custom code, with the integrations each project needs.
What’s included
- Webflow development
- Custom website development
- Custom interactions
- Performance engineering
- CRM & email integrations
- Booking & payments
SEO & AI search
We make your site easy for search engines and AI assistants to find, understand and point people to.
What’s included
- Technical SEO
- AI search & AEO
- Structured data
- Answer-ready content
- Core Web Vitals
- AI crawler access
Optimisation
We keep improving what happens after launch, using real behaviour data to turn more visitors into enquiries.
What’s included
- Conversion optimisation
- A/B changes
- Landing pages
- Copy reworks
- Analytics & heatmaps
- Monthly improvements
Recent work
Latest
The newest posts, across every topic.
Design
Planning and designing websites, and the tools we use.
Development
Webflow, Astro, Claude Code, CMS choices and what builds really take.
SEO & AI search
Getting found in Google and quoted by AI assistants.
Studio
How we run projects, from proposals to launch week.
robots.txt checker: can AI read your site?
Check whether ChatGPT, Claude, Perplexity and Google’s crawlers can reach your website, through robots.txt and your firewall, then generate a corrected robots.txt in one click.
Free, no sign-up. Takes about 10 seconds.
Who can read
From your robots.txt, and a live visit as each AI search crawler to catch firewall blocks robots.txt doesn’t show.
Your robots.txt
Generate a robots.txt
Every AI crawler named, so your choices hold. Paste it into your site’s robots.txt (in Webflow: Site settings → SEO).
Crawlers can reach you. Does AI recommend you?
Our free AI visibility checker asks AI your buyers’ questions, shows who it recommends instead, and why.
Why AI crawlers matter.
Before an AI assistant can recommend your business, it has to be able to read your site. ChatGPT, Claude and Perplexity each send crawlers to fetch pages when they search the web for an answer. If your robots.txt, or a firewall setting you may not know about, turns them away, you won’t appear in their answers, however good your site is.
There are two kinds of AI crawler, and they’re worth treating differently. Search and assistant crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages to answer a question right now: block them and you’re invisible in AI answers. Training crawlers (GPTBot, ClaudeBot, CCBot) collect pages to train future models: blocking them is a choice about your content, and doesn’t affect whether AI search can find you.
robots.txt and AI crawlers, answered.What people ask about robots.txt, AI crawlers and blocking them.
Enter your website above. We read your robots.txt for OAI-SearchBot and ChatGPT-User (ChatGPT’s search and browsing crawlers) and GPTBot (training), then visit your site as each AI search crawler to check your server or firewall lets it in too.
That’s your call. GPTBot collects pages to train OpenAI’s models; blocking it doesn’t stop ChatGPT search finding you, because that uses OAI-SearchBot. Many businesses allow both, so their content shapes what AI knows about them; some block training to protect their content.
GPTBot crawls to train OpenAI’s models. OAI-SearchBot crawls to find pages for ChatGPT’s search answers, and ChatGPT-User visits a page when someone asks ChatGPT to look at it. Anthropic splits Claude the same way: ClaudeBot for training, Claude-SearchBot for search and Claude-User for visits users ask for.
Yes. Cloudflare, for one, has a setting that blocks AI crawlers at the network, and some hosts block unfamiliar bots by default. This checker visits your home page as each AI search crawler and tells you if it’s turned away.
In Site settings, open the SEO tab and paste your rules into the robots.txt field, then publish. On WordPress, use your SEO plugin’s file editor or your host’s file manager; on Shopify, edit the robots.txt.liquid template.
Not a crawler, but a token in robots.txt that tells Google whether pages it crawls may be used to train Gemini. Blocking it doesn’t affect Google Search or AI Overviews, which use Googlebot.
Want AI to recommend you, not just read you?
Book 20 minutes with Matt and we’ll look at what’s holding your site back in AI search.
Book a call with Matt