Hi everyone, welcome to our videocast series with Qrator Labs. Today we are going to be talking about the web scrapping, but before that, please go and check our previous videos. This is the third video series that we are doing with Qrator Labs, and today I am joined by Dmitry Loshakov. How are you, Dmitry?
Hello, I'm great to be here, fine, what about you? I am doing good, and I'm excited actually to talk about something new this time, because last time I think that we did a good job with your colleague and then we covered as much as we can for the DDoS attacks and DDoS environments and what Qrator Labs offers in this field. But today we are going to talk about something else, we're going to talk about web scrapping and then the automated process of collecting data from websites such as texts, images or other content. This is what we call web scrapping, right?
And with the rise of AI, like anything else, this process has become much easier and more accessible to cybercriminals. Even people without the deep technical skills can now utilize these AI models and then the tools and the open source tools to build powerful and adaptive scrappers nowadays. There is also another side to it because AI is also making the scrappers harder to detect and then they can imitate the real human behavior and the traditional defense systems like bot challenges like CAPTCHAs or JavaScript challenges are becoming less reliable and then people always dislike them anyway.
All of this has to mean something, right? Yes, this means that we have to change our cybersecurity strategies and maybe look for tools that can help us with this new type of web scrapping attacks. So instead of static defenses, the organizations must rely on some new tools and the behavior analysis maybe or real-time decision making to identify and mitigate these malicious activities. I'm sure that Dimitri, you also have a lot to share with us on these topics. So if you allow me, I'm going to have a couple of questions on the topic.
But before we begin, I think that the most logical question would be, what is exactly web scrapping in your point of view and then who are usually its victims? Okay, thank you for your questions. It's definitely a hot topic these days.
I mean, the web scrapping, the AI and all this stuff. Let's start from the beginning. The internet is a gigantic library full of data and information. And as we all know, whoever controls information can monetize it. So in a few words, web scrapping is automated collection of the data from websites. It's important to know that the data can be collected for very different reasons and by very different crawlers, the tools that collect the data.
And on one hand, you have perfectly legitimized use cases, such like search engine indexing, large-scale analytics, and for example, academic researchers, data aggregation by authorized partners, something like this. On the other hand, malicious, unauthorized scraping. The goals are the same, to get data, but the methods may go beyond public access. Constructed user profiles provide APIs or unique content that companies have built up over years. And somewhere in between lies the gray zone, publicly available data that changed frequently.
For example, prices in an online stores or product availability. Many companies monitor their competitors' prices to adjust their own, or aggregators can collect product catalogs to offer better deals. All of that is technically public.
I mean, the data is public, but the data owner often wants it accessible to potential customers, not to competitors. And that's where it gets interesting. The line between acceptable and abusive scraping has become very blurry. For example, even the language models need a vast amount of data to train, and that data often comes from the open web. But what if I don't want my site to be part of this data set? A company may have no idea that its content is being harvested daily by dozens of automated agents.
From what you just told us, Dmitry, I think that these attacks are affecting our daily lives. It should be an issue for everyday people as well. But I know that we are talking about enterprises and organizations today, right? And with the use of AI, now everything is kind of affecting our daily life. But let's focus on some industries, I guess. And then I know that you've been dealing with such attacks as curator labs, as company, and then you have customer base from every region, from different regions and different industries.
But I'm sure you must have some data that is providing us some insights about which industries are affected the most. Could you maybe share us some information on this?
Yeah, of course. We published our quarter report several times ago, and you can check it. I think we can share the link with our conversation. But speaking by my own words, the first place is token by e-commerce. It's always e-commerce because Skyping directly impacts profits. Product descriptions, photos, reviews, and prices are stolen. I've seen cases where an online retailer lost a serious portion of its revenue because bots constantly pull prices and competitors use the data for dynamic undercutting. So direct damage. And it's not just general retail.
Online pharmacies, rental platforms, anywhere there is a product and price. The next point is betting and gambling. An industry where knowing your competitors' odds is crucial. Competitors collect odds, results, match statistics in real time. That data is then used for arbitrage betting or even to let bots place bets automatically. So in this sector, Skyping causes direct and financial losses. Then maybe I can tell about transportation, for example, and logistics, which includes the travel industry. Skyping here adds another layer of damage.
Not only stolen prices, but it disrupts dynamic pricing. I mean, when bots repeatedly query the same flight, the system sees artificial demand and raises the price. As a result, real users get distorted prices, while ticket aggregators may show lower ones. Next finance. Attackers hunt for publicly available, but sensitive data.
Trades, traded software, corporate contacts, sometimes they even try to access data inside their personal accounts. And let's not forget that scrappers' used data often finds itself in financial staffing and brute force attacks, which use essentially the same software automation stack. So it's a vicious circle. And last but not least, media and content platforms. Imagine an online newspaper publishing an exclusive article, and a few minutes later, the same text appears on shady, semi-legal websites that steal your traffic and add revenue.
So from what you tell us, Dmitry, I understand that the problem is more like the economic than the technical problem. Because you tell us that the dynamic numbers of the betting industry or the dynamic price changes of the travel industry, I think that the problem is different than what I would expect. So could you maybe clarify this a bit more?
Yeah, you're right. Scrapping is a tool of competition, and the problem is more economic than technical. It can destroy your pricing strategy, steal content or increase infrastructure costs, of course, because all these pseudo-visitors still create real load. But I don't want to say that it's a big deal. It's not a DDoS attack. Your server continues to be available. And so this is the scariest part. Most of these attacks go unnoticed. To the naked eye, everything looks fine. The website works, the page is open, but the data is already gone.
Yeah, I understand. So I think that we kind of covered what it is. But for those who are interested to learn what is changing with the arrival of AI, what would you like to say about it? Like what is exactly changing with AI to web crawlers?
As usual, with the AI, the biggest change is accessibility. So AI lowers the bar. In the past, scraping requires technical expertise. Now anyone can build an intelligent scraper, just open and charge GPT or Cloud or something like this and type, okay, give me the data from this site. Collect it, pretend as being a real human and move the mouse and rotate browsers and so on. That's it. You get working code that behaves like a human. AI doesn't just write code, it explains it. How to bypass protections, if the site returns an error. It can even debug itself and suggest improvements.
What used to take weeks now takes hours. So there are far more attackers and their tools are much more sophisticated. So you mean like they're smarter? Ironically, the attackers became smarter by making the scripts more human-like. Modern scrapers powered by large language models and autonomous agents don't just send HTTP requests. They simulate a real user, moving the mouse, scrolling, hovering over elements, even adding missing clicks on purpose. And this happens dynamically, not from a static script.
If they encounter simple defense like a server-side JavaScript challenge, the bot can read the code right on the page, open a VM or something like this, execute it like a browser and get the token and continue. If that bot meets capture, it either calls an open source tech recognition tool, some OCR or something like this, or even uses its own neural model to generate hints for a human solver, for a human operator. Each month, these agents get better and faster. So you mean traditional defenses don't work anymore? Should we understand this? Exactly.
Classic tools like CAPTCHA, server-side JavaScript challenges or connection fingerprints like GA3, they're outdated. You're no longer fighting static scripts, but adaptive agents. If one method fails, they try another. That's worse. These agents are now invisible for most anti-fraud systems. They pass behavioral filters because they act like people, like real people. Random delays between actions, realistic timing, mouse trajectories that no longer start at pixel zero zero from the left corner, left up corner. From a lock or anti-fraud perspective, it looks perfectly human.
So then we can say that we are already in the area of invisible scrapping then? Yeah, absolutely. In the past, you could spot a bot by its tool set or behavioral pattern. Now it's much harder, especially without deep session level analysis. And since real users behave very differently from each other, AI-powered bots can even look more average than humans do. It's a paradox because the humans start to act more unlike than humans do before. The better you defend, the higher the risk of blocking real customers.
It's a constant can-and-mouse game where false positives and false negatives are inevitable. Your strategy now is a balance between rigidity and accuracy?
Yeah, I think that now that I would like to interrupt you because I think that you have already explained us what it is and how it's been enhanced by the use of AI tools. But I think that we should also talk about how we defend ourselves, right? And maybe I could ask you how we can defend ourselves against such smart scrapers and what kind of tools, what kind of solutions actually work today. And maybe you could also give some examples from your solution from Curator Labs as well.
Okay, the first thing is to accept the simple fact that if information has value and it's online, it will be copied. Definitely, no exceptions. The strategy's job is not to make it impossible, but to make it unprofitable. In other words, make scrapping so slow or costly that the copied data becomes outdated before it's even usable. Or rate the cost enough that the attacker's profit margin disappears. So here are two principal tasks the defend needs to solve.
The first one is being fast and efficient at churning down the bulk of unwanted and malicious bots on their easiest, earliest interaction with the system, which is the initial request to the server. This is what we at Curator Labs focus on. The second is being able to detect advanced bots soon enough, sooner than they will cause any harm to the system or other users. Service-side JS challenges or GA3 fingerprints are outdated and easy to fake. To be fair, anything can be spoofed, but the more things an attacker has to fake, the higher the chance of a mistake.
So we are still able to make the scrapping stack itself extremely hard to build. That's why we use client-side JavaScript challenges, not server-side, not just as proof of work but as proof that our script actually runs in a real browser, not in a headless one or not from a terminal.
Well, I mean, as also an internet user and also a cybersecurity industry analyst, I also am not a huge fan of both challenges and now I like to see more and more invisible challenges. But I think that we still need those challenges, although it changes the user experience. But you could maybe tell about how your solution is affecting the user experience, if the new methodologies are making it better in the user experience or maybe making it worse. How would you comment on this? Fortunately, the main thing that breaks the user experience is captures.
And we didn't use the capture, so our protection system didn't interfere with the user experience at all. Modern devices handle this instantly. The script executes so quickly that the user never notices it.
I mean, the JavaScript challenge. I got it. What about the second approach you mentioned earlier? If the data is valuable enough for attackers to use forms or real devices with actual browsers or even human operators, then we need to go deeper into intent analysis.
So I mean, a real device with a human operator cannot be marked as an automated bot device because of a human. That's why we need to go deeper. And the goal is to slow down or frustrate the attacker's process without harming real users. As a first filter, you can use capture.
Yeah, we talked about it. But not the text ones, not these ones that need to put the word with some distortion. Modern bots solve this better than humans. And use interaction-based capture, those that analyze mouse movement, timing, and natural human interaction patterns. Capture definitely will take some time, but it can be run smoothly while you're reading the web page you're opening. And this capture is something we're actively working on at Curator Labs, and we wanted to publish it in the near future. But not everything can be outsourced.
Because some forms, business metrics, and data are often too unique or protected under NDA. That's why some defense tools work better inside that exact infrastructure. For example, server-side AI can analyze thousands of parameters, from mouse movement patterns to click density on different UI zones. This leads to what we call real-time risk scoring, assigning each session a dynamic risk level. If a session looks unusual, but still faces challenges, don't block it, just monitor it. If the risk increases, escalate, delay responses, or block entirely. Oh my god.
So what I understand is that static checks alone are no longer enough then? Unfortunately, no. Not when the data has real value. Running a farm of real PCs with humans behind them is expensive, but sometimes ROI justifies it. For complex cases, there's intent-based detection, understanding why a user does something. If someone opens 500 product pages per minute, even with natural clicks after a JS challenge, that's definitely not a real shocker. The key is that protection must learn. And if the model sees a new pattern, it updates itself.
The final stage, defense becomes self-learning, just like the attack. So AI is now actually utilized both in the attacking and the defending side, but would you like to maybe also give us a couple of examples of how is it used in the defense strategies?
Yeah, exactly. AI can be both the poison and the cure. AI today analyzes user behavior, correlations between millions of sessions, and even predicts attacks before they start. It detects micro-patterns that humans can't see. For example, that both sessions switch pages slightly faster than a real user, or that clicks happen with a constant or too short delay after page load. Those differences are tiny, but at scale, AI picks them up easily.
Of course, efficiency of AI-driven defense depends entirely on data quality. If your model is trained on a small dataset or traffic known from your different site, it will make mistakes and false positives. So the takeaway should be that the companies must build expertise and deploy AI-driven countermeasures then. That's what I would say.
Yeah, me too. That's what I wanted to say. Fortunately, these attack tools still require skills and infrastructure. They are not cheap. And since there are plenty of unprotected targets out there, attackers don't yet have much incentive to evolve. But we're already in a world where, when needed, attacks become intelligent. Scrapping is a simple yet powerful instrument of strategic pressure. By copying information that's already public, AI just lowered the barrier and made it adaptive. It can now observe, analyze, and react faster than a human ever could.
But the same advantage works for defenders. If attackers have smart automation, defenders have smart detection. The battle has moved from code to data and learning speed. Whoever recognizes patterns first, wins. The mission of cybersecurity today is twofold. Build a wall high enough to filter the noise and build a system that understands who is coming and why. This is no longer a human versus scripts. It's an AI versus AI. And whoever teaches their AI to think faster, keeps control of the data. I think it was a great closing part.
I think AI versus AI is going to be something that we are going to be hearing for a long time and no matter what the topic is. But yes, today we tried to cover what web scrapping is and how it was before AI and then now what has changed with the proliferation of the AI tools and if it is more difficult to mitigate them now. Dimitri shared their expertise and what Curator Lab offers in this field with us. It was great to have you here, Dimitri. I'm looking forward to continuing with this series. I think it's really fun.
I also learned new things from you, especially when you share data from your end user side, from your customers. I think it was a good discussion. So looking forward to having you here again. Thank you for having me, Oskar. Thank you so much. It was a real pleasure talking to you. Stay secure and goodbye.
Thank you, Dimitri. Bye.
