Getting Structured Data from Web Pages with AI and Re-usable Specs

As I say from time to time, every now and then I build something as a proof of concept and the result is… ok? I’m not really sure if the idea actually made sense and if the final result is usable, but the exploration was interesting enough that I think it warrants sharing with you good people. That’s what today’s post is. This demo all began with a simple idea – how do you handle getting RSS like content from a site that doesn’t actually have an RSS feed? This typically relies on looking at the page as data to be parsed. For a while now, there have been multiple good ways to turn a web page into structured data. As an example, AgentQL lets use a GraphQL-like... more →
Posted in: JavaScript

Extracting Data from Web Pages with AgentQL and BoxLang

I discovered AgentQL a few weeks ago and have been thinking about it quite a bit. In a nutshell, it lets you perform queries against a web page. They’ve got a simple query language that kinda reminds me of GraphQL, but simpler. So for example, consider the page you are on right now – if I wanted to get the tags, I could use this query: { tags[] } And it would return: { "tags": [ "#development", "#boxlang" ] } What if I wanted the links? I could change my query to express this: { tags[] { label url } } And then get: { "tags": [ { "label": "#development", "url":... more →
Posted in: JavaScript

Using Generative AI to Parse Web Pages into Data

A few months back, I took a look at using JSON-LD to turn a recipe web page into pure data: Scraping Recipes Using Node.js, Pipedream, and JSON-LD. This relied on a recipe actually using JSON-LD in the header to describe itself, which is pretty common for SEO purposes. Still, I was curious as to how well generative AI could solve this problem. In theory, this could be a good ‘backup’ in cases where a site wasn’t using JSON-LD and a general exploration of ‘parsing’ a web page into data. I’ll be using Google Gemini again, but in theory, this demo would work in other services as well. Here’s what I found. Converting a Web Page into Structured Data In order... more →
Posted in: JavaScript