@ireneista yeah, the search engine that i'd like to build Eventually(tm) would be one that has a really locked down set of allowable domains and paths. honestly something like a web of trust model might be good
@astrid we'd also ideally like to do it in a way that gives everyone a copy of the search index and all software needed to produce it
Replying to @ireneista@adhd.irenes.space
@ireneista i have done a small amount of research into how this could be done. you can definitely go with https://commoncrawl.org/ to start out with, instead of crawling sites yourself.
i feel like the governance and funding is the bigger issue, though. search engines become shit when they become ad engines. if i were to operate something like this, it would either have to be a money sink hobby thing i give only to friends, or an explicit "pay a subscription for access" type thing.
commoncrawl.orgCommon Crawl - Open Repository of Web Crawl DataWe build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.i feel like the governance and funding is the bigger issue, though. search engines become shit when they become ad engines. if i were to operate something like this, it would either have to be a money sink hobby thing i give only to friends, or an explicit "pay a subscription for access" type thing.
Replying to @astrid@woem.space
@astrid
neato!
I've been cobbling together a little toy search engine that will be based on my personal browsing history and bookmarks, cuz I didn't know things like this existed. I was going with history and bookmarks, cuz crawling is harder since the AI nonsense, even when you're doing it politely, and my rural internet is sloooww.