That’s going to be a lot more work since comments and posts are decentralized here. You can probably easily get some of it but it will be hard to get all of it.
It’s actually even easier than that. Instead of setting up an tool to make up requests for the API, you can just set up a bridge that will dump everything right into your database. The wonders of federation.
If you can set up a Lemmy instance and apply a little elbow grease to manually follow a few instances, that’s pretty much all you need to have the data come in automatically. You’d probably need more knowledge about how to actually get the data out of the DB than the initial setup, which could be done by somebody just copying and pasting text.
Long Live Lemmy
Well, they can (and will) still scrape us if they want. Just nobody’s making a buck off of it.
yet
That’s going to be a lot more work since comments and posts are decentralized here. You can probably easily get some of it but it will be hard to get all of it.
It’s actually even easier than that. Instead of setting up an tool to make up requests for the API, you can just set up a bridge that will dump everything right into your database. The wonders of federation.
If you can set up a Lemmy instance and apply a little elbow grease to manually follow a few instances, that’s pretty much all you need to have the data come in automatically. You’d probably need more knowledge about how to actually get the data out of the DB than the initial setup, which could be done by somebody just copying and pasting text.
All better than that piggyboy getting free money
Why?
The reality though is I can train LLMs off Lemmy data all I want and I don’t have to pay ANYONE a dime…