Scrape data from URLs in database

[realscapemedia]

I’m not a coder. I’m just trying to figure out how to properly create an automation that will take a url from one cell, scrape some data from the webpage located at that url, and then write the return into another cell in the same record. Then do that for every record in that database. Can someone help me out?

[Thea]

Hi @realscapemedia, welcome to the community!

That’s a solid use case. Here’s a simpler version of how you can set it up:

You can use either the “Run Script” action or the “Send HTTP Request” action to fetch data from a URL, then write that data back into a field in the same record.

Here are two helpful tutorials:

How to retrieve the data received by the "Webhook Received" trigger in downstream Actions?

The “Webhook Received” trigger is an advanced feature that requires specific configuration. Here’s a step-by-step guide to capture and use its output data, using an example where data sent via Postman is stored in a database via Bika Automation: [image] Configuration Steps Add the Trigger: Create a new Automation and select the “Webhook Received” trigger. Click “Save” to generate the unique webhook URL (this URL is only created after saving). [image] Enable the Automation: Activate …

How to Batch Create Records Using the "Run Script" Action

When importing external data into Bika’s database, many users initially attempt Automation workflows combining “Send Http Request” actions with Loop + Create Record actions. While this approach works for simple data transfers, it reveals limitations when you need to: Clean raw data from external sources Transform formats (e.g., date normalization to ISO 8601) Implement complex business logic Handle API error cascading This guide provides a script that shows you how the Run Script action empow…

[realscapemedia]

Thank you for your response. I’ve been trying to make this work for a couple of days, but I’m not being able to. My JS script keeps returning an error that the URL is invalid. I’m not sure exactly what I’m doing wrong since I can’t seem to get any console output besides that.

I don’t think what I’m trying to do should be that complex. It’s just:

1. Get URL from database record
2. Scrape data from that URL
3. Write scraped data into another cell in the record
4. Iterate for all records in the database.

Any ideas?

[Thea]

Hi @realscapemedia, that really depends on your use case, but you might find the Community Manager Agent template helpful:

Discourse Community Manager

It comes with an MCP server called fetch, which can pull data from URLs and convert it into markdown. It might be exactly what you need for this scenario.

[Chowderr]

Just curious on what you are scraping?