our preference is always to find out why the block is happening and try to convince people it should be otherwise; widespread abuse of robots.txt does no-one any good, having been crawling and indexing for so long it’s a standard that we understand and are quite fond of

we can see some of the perils and pitfalls of it too, but web builders need to be given some tools and assurances that those tools will work for them

IP already hits a wall, also better to not get a reputation as a bad bot, it’s taken a while to get known for being friendly and respecting rules, to us you should follow robots

We make money from our API, what they’re referencing is a beta ads programme which was running

Agree to disagree here, but I’ll refer to Cory Doctorow for a contextual vs behavioral/tracking ads comparison, one which is very good: https://pluralistic.net/2020/08/05/behavioral-v-contextual/#contextual-ads (applied to the media, but the general thread is relevant)

which language are we talking?

Reddit doesn’t allow us to crawl: https://www.reddit.com/robots.txt

qwant is bing, mainly

we make money mainly from our api, our investors are patient private capital and we don’t take vc, appreciate your point but these are fundamentally different situations, our ads (when they run) will also be contextual so more of a ddg situation than a “makes users into products to be sold to advertisers”

fair enough if it’s not for you though

this time, it will be different /s

The Express? There’s definitely a not-reading-it option

forks of firefox still keeping things going such as mullvad browser, waterfox, librewolf

Thanks a lot, I’ve raised the Kdenlive query. When it comes to searching for stuff with “yt-dlp” you mean there’s normally another phrase or words in the query? Do you have an example; understand the problem but just trying to raise it with as much of a solid example as possible 🙏

thanks for flagging this, will get it raised

Got any examples of searches you could do with seeing improvement on? A big part of our process is collecting these (as well as identifying issues in crawling off the back of them)

plz do not sue us

this comparison

the table you’ve cited is very out of date when it comes to us and other inclusions

being backed or debated at least by UK politicians

We came up in Hansard yonks ago 2011, mentioned by one MP

they do have individualized answers as well

I don’t quite follow but if you mean results are personalised can I ask where you got that information? One of our main things is that we don’t affect results based upon much more about you than country-level boosts

thanks for mentioning us; Qwant also uses Bing (same as DDG) so if they don’t have a personal parameter then it should be similar. That being said we (obviously) think people should give Mojeek a go :D

can’t emphasise too much that this piece is a very necessary read for anyone who wants to know about search; not just because it says good things about us, but because of the depth of research which has been put in here. Most times you encounter an article about indexes they are just taking whatever a (meta)search engine says about themselves, not even looking at privacy policies for “relationships with microsoft” etc. or doing any comparative work.

yes we are indeed a standalone, the sizes of bubbles are reflections of the number of index-usage relationships yep is attached to ahrefs: https://techcrunch.com/2022/06/03/yep-search-engine/ where I can speak to where results come from, I couldn’t tell you about their model

we’re working on it 😉 slow and steady and all that; we also fixed a bug with recrawl recently that should be improving things