I’m strongly against AI uses like these tbh, and it’s really no use testing their ‘intelligence’ on information that’s readily available online because that’s kind of just… what it does? It scours the internet to answer questions you ask it. I really don’t see the difference between using AIs like this and just… running a smart Google search yourself? If you know what words to include and exclude when searching about specific topics (I live and die by using quotation marks and -excluding words from search results) you’ll end up with the same knowledge as an AI chatbot can give you.
They’re really not very smart at all beyond that point, as people have shown through various exploits you can use to get it to give you information (again, readily available) that it refuses to otherwise, like how to make explosives etc.
They cannot answer questions about literature beyond facts or what’s already been said on existing websites for example, and they also can’t access articles that are behind paywalls, including things like academic journals, so askng it more technical questions is useless, and some have been shown to completely make up information when the only facts are locked inside paywalled journals.
That aside, I also don’t really care for character chatbots like character.ai either, and I just don’t see a point? They all run on the same knowledge base, so there’s only so much variance they can achieve when speaking as different characters, and it can never be accurate unless trained heavily on relevant material, like (IIRC) the Mario bot. Maybe I’m just picky, but I went in there going “he would not say that.” Again, I obviously know it’s because they run on a single knowledge base, but the inaccuracy takes away from it heavily for me. I really don’t see the fun in it myself, but I’m just very pretentious about everything lol
In conclusion: These are just fancy Google search bots, and I’m heavily unimpressed by character.ai and how surface-level it feels.