Page 1 of 1

[REQUEST] Server side url tittle fetcher (url tittler type)

Posted: Thu Aug 24, 2017 9:58 pm
by HeXiLeD
The purpose of this module is a server side implementation of what irc bots and some irc clients do every time somone pastes a url on the chatroom and the irc bot fetches it's tittle.

Example: the unreal forums bot @ #unrealircd

Why ? Many modem chatroom platforms offer this service from the server side. One example is "matrix" another is "facebook". Soon as the url is pasted, the tittle and page preview. On irc we can't see the preview but we could see the tittle and avoid an irc boot and scripts and the server would fetch the tittle.

Module name: url_tittler.c

This type of idea should be a default feature at some point. Lets not fall behind.

This would be a channel mode implementation and the output could be sent as a channel event similar to join, part,quit, etc but this other request must be taken in mind:
Server side ignore channel events (join,part,quit,etc)

Re: [REQUEST] Server side tiitle fetcher (url tittler type)

Posted: Fri Aug 25, 2017 6:44 pm
by Gottem
I don't know Matrix at all, but Facebook is an entirely different platform. I haven't looked into it, but I'm pretty sure it uses an XMLHTTPRequest (or something similar) in Javascript to load a given URL, get the title and a thumbnail and modify the chat's HTML to actually display it.

It should be done by bots, not Unreal. My reasons for it:
  • The call to load the webpage would (by default) block all other activity. So if a webpage is slow as shit and takes 20 seconds to load, all messages sent in the meantime will be delayed by that amount of seconds too. Unreal is single-threaded but uses various syscalls to figure out what socket/client sent/should receive something.
  • Seeing as Unreal is single-threaded, you might think making the module multi-threaded would work. But, at the end of the module's logic, it'd still have to wait for the thread to finish (pthread_join() for example). So it would end up blocking the rest anyways. If you don't use the join() call the resulting behaviour is undefined, so it might simply get killed once the module returns to the core code.
  • Maybe forking then? No can do amigo. For one it's ugly as hell to fork for every URL. Secondly, forking creates a new process which inherits a copy of the current memory contents (variables, module code, etc) which gets really inefficient fast. Unreal is supposed to be memory-friendly and efficient.
That is, unless Unreal has some super secret functionality that I missed while grepping the source. :>

Sidenote: you can't just make up "channel events". They're all defined in the standard and IRC clients interpret messages based on that.

Re: [REQUEST] Server side url tittle fetcher (url tittler type)

Posted: Sat Aug 26, 2017 12:25 pm
by HeXiLeD
How about with a bot feature added to botserv or would this be still part of the problem described above ? The main goal is to avoid using a regular bot.

Side note: Would you say that this feature would be a lot more effective if implemented by default on irc clients? I wonder if it is how the matrix riot and other clients do it.

Re: [REQUEST] Server side url tittle fetcher (url tittler type)

Posted: Sat Aug 26, 2017 12:40 pm
by Gottem
BotServ isn't part of UnrealIRCd, it's part of Anope services. I don't know how it would work for that as I'm not too familiar with Anope's APIs.

And yes, this should ideally be implemented client-side so everyone can decide for themselves if their client should show the title (only to them and optionally with a thumbnail). A lot of clients already support this.

Re: [REQUEST] Server side url tittle fetcher (url tittler type)

Posted: Sat Aug 26, 2017 1:26 pm
by HeXiLeD
Yes there a lot of clients do support this feature, but it causes a security problem as anyone (attacker) can just past any url (looking like harmless) to the chatroom to have clients using their auto url tittle fetcher and phish the client real IP address for all sorts of nefarious purposes.
So far I have not seen this client side feature with proxy (socks5) functionalities.

Re: [REQUEST] Server side url tittle fetcher (url tittler type)

Posted: Sat Aug 26, 2017 2:10 pm
by Gottem
Not a lot we can do about clients not proxying those requests too. ;] There are tons of other ways to get someone's IP anyways, like if they click the link and their webbrowser isn't proxied.

Also, Anope generally runs on the hub so you'd be exposing that IP which you generally don't want to do.