Gosora - Supremely fast forum software

Discussion in 'Forum Software Development' started by MagicalAzareal, Apr 30, 2019.

  1. MagicalAzareal

    MagicalAzareal Magical Developer

    I'm back here after I dunno four years, I can't remember, my perception of time is too awful. See my introduction thread for more details on who I am.
    I've been experimenting with Go, big thanks to Euan for pointing that language out and I threw together a bug tracker which morphed into a custom forum software.



    I was inspired to create it by the RR scaling incident, a lack of good systems which met my needs, a lack of systems which could keep pace with the ever changing world and, most importantly:
    To learn a new programming language and to experiment with more low level technologies in a web context (I'm slowly digesting the knowledge I got from reading the HTTP stack implementation Google wrote for Go).

    It has a bunch of features which are essential to me, but probably more boring for others.

    * Live dashboard showing me how much CPU, RAM, etc. is being used on the system. I love this feature.

    * Live Notifications when someone does something, because that's kind of important to me. Also, some things like the topic list are live, but I never quite got around to rolling that everywhere.

    * I try to keep the stack simple in true Go style. There is no requirement to have Node installed or Redis or Imagick or any manner of things which complicates operations. Cron jobs or other task schedulers aren't required, as the main process handles tasks and what-not itself.

    * Zero cost analytics (I cheat by holding the data in memory and batching them to the database every now and then).

    * Zero queries on hotpaths (pull random things out of memory).

    * Asynchronous programming without any of the pain (except for data races, I hate those x.x).

    * C / Rust level performance from Go absent GC woes and the standard library over-allocating (Go 2 is the magic word usually used for problems like that).

    * Experimental transpilation to client JS and server-side Go. This one was partly for fun, but it works well too.

    * I avoid C and Assembly for security reasons where possible. I might not be able to sleep from the ensuing paranoia, I'll see if I can introduce some more when I add support for Docker and am able to safely isolate those elements away. Maybe process level isolation might be enough?

    * I leave out a fair bit of fluff which I think distracts from the core experience.

    * Can probably be run off a toaster. My laptop with constantly maxed out memory usage from Firefox, Opera and Chrome waging war on each other is probably a good example, but there are also smart toasters out there these days.

    * Four default themes. You can switch between them via the theme selector in the footer. Nox is the defaultest default and Cosora is the most advanced light one.

    * More in https://gosora-project.com/topic/features-introduction.72

    The analytics includes route counters, UA agent counters, referrer counters, memory usage tracker (coming soon), etc. and has graphs plotting out this data over time.

    I take care to anonymise all of the data which I collect by aggreggating all of the data together to make sure that the privacy of my users is adequately preserved, not just because "GDPR", but because privacy is a fundamental human right.

    I plan to expand on that by periodically purging request logs (standalone), IPs associated to posts / users, etc. But that'll probably be a setting of some sort.

    Very crude benchmarks show throughput over a thousand times faster than MyBB which on it's site claims to be "one of the fastest forum software", although it wasn't anything particularly scientific and will likely vary depending on the environment.

    MyBB was picked as my guinea pig as it was always my favourite software (even if I've outgrown it now) and it wasn't as difficult to setup for tests as phpBB. I also tried NodeBB, but it short-circuited on a too many requests error page, so I couldn't get it's true throughput, but the error page wasn't too performant.

    It can be run as a standalone server or proxied behind Nginx or whatever fits one's needs.

    I do have to note that I might be adding support for HTTP/3 soon which may not be compatible with Nginx and the like, but which does improve client performance by a fair bit, so things might get complicated there (or just turn that off).

    Future stuff:

    * Finish adding ElasticSearch, or find an equivalent library for Go, to reduce the amount of ops work I have to do.

    * Finish hardening it. I am paranoid about security, so this is a task which basically never ends.

    * Implement "social" stuff like friending and maybe customising profiles?

    * Support databases other than MySQL and the partly broken MSSQL.

    * Add a more sophisticated updater than the one I have now.

    * Figure out how we can beat AsmBB, if we don't already lol

    * Move gosora-project.com to a better host.

    And before you ask. It does not run on shared hosts.
    Well... Someone managed to get it to work on one, but I don't advise it.
    Last edited: Apr 30, 2019
    • Like Like x 7
    • Pure Genius! Pure Genius! x 1
    • Appreciation Appreciation x 1
    • List
  2. MagicalAzareal

    MagicalAzareal Magical Developer

    I forgot to mention something very important. There are four default themes.
    You can switch between them via the theme selector in the footer, so if you don't like dark themes (Nox is the defaultest default), then you might want to take a look at Cosora :)
  3. MagicalAzareal

    MagicalAzareal Magical Developer

    Some things I've done since April 8th.

    I'd like to eliminate server rendering for users with JavaScript enabled entirely (this'll be a fun one, it's easy, but making it performant will be the real challenge as always lol), periodic IP purging and maybe HTTP/3.

    HTTP/3 is slightly annoying though as it involves opening an extra port, which requires an extra listener and it's incompatible with just about every CDN making it useless on any serious site. Even Cloudflare, the most advanced CDN, has yet to roll out support for HTTP/3.

    CDNs are very very important, much more so than HTTP/3.
    I've seen users on some forums suffer from eight second load times when it could have been a fraction of that, because the admins were too stubborn to use a CDN.
  4. cornnfedd

    cornnfedd Captain Futurama

    Wow that is super fast!!
  5. MagicalAzareal

    MagicalAzareal Magical Developer

    Thanks, I've been doing loads of client optimisations to get things faster on mobile and particularly in places like Australia where pages can take an annoyingly long time to load.
  6. MagicalAzareal

    MagicalAzareal Magical Developer

    I've done a few things, although I've been a little depressed (alright, not a little), so a bit less than I would have liked.

    Some of the things I say might go over your head, if you don't have a certain amount of knowledge about computer science, so I'll briefly explain some of the basics.
    Just stuff you'll get from skimming any random C++ tutorial. I might skim over some of the finer details, so please don't bite me lol

    Computer memory is a giant grid of sorts. Each slot of sorts has it's own memory address.
    Data is stored in these slots and you can access these bits of data via simply named things known as variables.

    There is one type of variable however called a pointer which instead of holding a bit of data in that memory slot will instead hold a number pointing to another memory slot.
    Now, you might be wondering. Why would you do that?

    Well, instead of making copies of the same thing over and over in a number of slots, you can just point a bunch to the same slot and reuse the contents in lots of different places and repoint elsewhere when you want to change whatever is backing it.

    Basics taken care of. Onto the actual changes.

    I've added a simple spam filter to deal with the spammy referrer bots, as they were becoming a bit of an eyesore. You can still click on the filter to expose them.
    This one was snatched from a lovely dictionary which I'm very fond of, but you can now do site searches from the URL Bar.

    I added a couple of analytics panes to plot the amount of memory Gosora has used over time on a nice little chart. Go tends to request more memory than it needs from the OS, so there's a chart showing actual memory use and the amount of memory that it has reserved.

    I also improved the UI for the debug page, it's largely for debugging the program, so it may be hard to follow. And yes, I do need to upgrade the database on that specific local build.

    I'm also experimenting with a small memory reply cache. Not much to see here, as I'm still collecting data to see how well it does.

    I've implemented meta store which is a small key-value store, just for simplicity in a few areas, like handling daily tasks.

    Topic / Reply / Profile Reply IP Addresses are now purged from the system after 180 days elapse for privacy reasons. You can configure or disable that in config.json, there is more to come here.

    I've added a couple of provisional exclusion flags to the analytics panes. You currently need to know the right URL parameter to make use of this. This was mainly added so I could reduce the amount of noise on the routes / languages analytics panes, as one particular route / language would tend to dominate the charts.

    I have also made some efforts to dramatically reduce the amount of bandwidth being used in the steady state.

    I've also fixed a number of bugs including one where the WOL widget would wake up too frequently to rebuild itself and trigger associated plugin hooks even when nothing had changed.

    I'm doing a rewrite of the hashlink, mention and URL parser right now.

    So far, I've dramatically reduced the number of edge cases (over 70 new test cases), improved performance by reducing the number of memory allocations and reduced the data moving between the CPU and main RAM by using the same pointer for Content and ContentHtml when the two are identical.
    • Like Like x 1
    • Informative! Informative! x 1
    • List
  7. SAFAD

    SAFAD Developer

    Hi, this is a nice project to see especially in a market dominated by PHP based forum software, kudos on choosing Go.
    A couple of complaints I have after taking a quick look over the source code (take them with a pinch of salt since I didn't do an extensive analysis)

    1. the installation process could be changed into a way better system (similar to how wordpress and other php based CMS/forums do)
    2. You do not use Package versioning/Dependency management (most popular is the "built-in" dep and Govendor, I have been out of the Go scene for a while now so I am not really up to date with this) which is considered "bad" development paradigm in modern days.
    3.File Hierarchy is quite confusing, I suggest splitting this huge project into packages and sub modules and structuring the project in a way the root directory is cleaner, yes this doesn't matter because in the end a "single compiled file" does everything in a production environment, however to ease contributions and developer ecosystem growth, this needs to happen.
    4.For databases you are using direct SQL queries to manage the database, this is fine as long as you intensively know how to handle the database efficiently, but in an age of Agile development and Laravel it is more intuitive to have an ORM instead of using this approach (steer away from GORM, that library is bad and I regret ever using it).
    5.Again on Databases, if your approach is to avoid third party libraries and dependencies to handle database, I suggest to implement a system that provides two APIs (One for Direct Queries, and one for ORM) based on the "database/sql" directly and not using the drivers provided (driver creation is quite easy in Go) as those drivers will add an extra layer of third party dependence to resolve issues etc.. which brings us back to to my second point.
    6. (take this one with a kilogram of salt as i haven't fully checked) it seems your plugin system requires a whole software recompile to enable them, steer away from it and use "Library load approach" (I think i forgot the term, but its something similar) as in plugins are independent from the software itself, and extend it externally while being built using the "-buildmode=plugin" to provide .so (maybe also .dll for windows?) library that can be loaded in runtime with the compiled software.
    7.Go's main selling point isn't only the speed, the ease of use or cross-compiling, its also real time data flow (or whatever they call it these days) the forum uses this intensively in the dashboard (stats i assume), however it would be a great selling point to your forum if you use this even in the forum discussions, categories etc etc (figure out how to not confuse the hell out of the user with new data/posts being loaded in front of him, simply a smooth transition to add new data to the displayed UI).

    Overall the project seems quite nice and built on top of sane technology, I commend you on that not many use Go's capacity to its fullest extent.
    I suggest you advertise the project in Go channels like Slack and you'll get tremendous help and suggestions to build a reliable forum software.

    P.S: I think I have talked too much to the point I seem to be criticizing the software, if this is how you feel I'm sorry, it was never my intention, just what I think of the project hoping you can see my point of view and improve the software, Forums landscape needs this, Gophers need this, the world needs you!

    Best Regards
  8. Lisa

    Lisa Chaotically Proportional

    Looks very slick. Well done and congratulations!

    Have to admit though
    This made me smile :ROFLMAO:
  9. Oedipus

    Oedipus Participant

    Themes could do a little improvement.

    Anyways, I love it. :)
  10. MagicalAzareal

    MagicalAzareal Magical Developer

    Thank you for the kind words ^^

    Not much has happened, been busy busy busy, but I have made a bit of progress on the conversation system (ala private messaging), although I haven't made that available in the softwate yet.

    It'll have a setting which allows you to encrypt all the messages in the database with AES, so that even if the database instance / database server is compromised, they will also have to compromise the application to gain access to them. I might also have to encrypt some user elements to make this work completely.

    I also considered adding end-to-end encryption, but it's very likely that users will lose their private keys, so I'm not sure how I feel about that. I have a good idea of how I would implement it while retaining the ability to report inappropriate messages without breaking confidentiality.

    A bunch of things like favicons have been fixed and admins can now change other users' avatars, so if you ever want to vandalise the profile of someone you hate with a very childish prank, then the world is your oyster, and maybe you can remove an inappropriate avatar or two while you're at it.

    The debug page has also been growing and I'm looking into pushing the segments out one by one while the queries run in the background with a bit of HTTP magic. No JavaScript or any of that nonsense. We'll see how that goes.

    The debug page now has a load of data like how much disk space your attachments are using, how many rows there are in a number of important tables, how many records you have in cache, etc.

    I've also made a number of small changes to improve performance here and there and I merged in a pull request from asminozhka to speed up compilation a bit. There's probably other stuff I missed too.
    I use the official one. Go modules.
    That only works on Linux. If you're using a BSD (including OSX) or Windows, then you're pretty out of luck, although they do seem to be making progress in those two.

    And thank you for the feedback, I'll get back to that more indepth later.
  11. MagicalAzareal

    MagicalAzareal Magical Developer

    Not much in the way of progress over the past few months, I've been a little busy.

    I'm currently working on a simple conversations system, you can see some of the pieces falling together in the repo. I also hardened the software against a certain side-channel (I'm not sure this one is possible to exploit, but you can never be too safe).

    I'm also trying to save some bytes here and there, refactored some things to reduce the amount of boilerplate, added a couple of privacy settings to disable referrer tracking into and out of the site (I know some of you love privacy). optimised some things and fixed a number of bugs.
  12. \o/

    \o/ an oddity

    I find software that is named after the chosen programming language rather shady, to be honest. A programming language is not a feature.

    Instantly found a bug: Everytime I change the forum theme on gosora-project.com, the theme selector's sorting is different.
  13. mysiteguy

    mysiteguy Administrator

    That's rather nick-picky considering phpBB is one of the oldest open source forum software packages still maintained with a fairly large install base.
  14. \o/

    \o/ an oddity

    I find phpBB rather shady.
  15. haqzore

    haqzore Devotee

  16. \o/

    \o/ an oddity

    Please move discussions about me and my intentions over to the personal message system.
  17. haqzore

    haqzore Devotee

    Sorry, allow me to clarify.

    What, aside from the name, do you find shady about Gosora?
    What, aside from the name, do you find shady about phpBB?
  18. \o/

    \o/ an oddity

    Gosora: So far, only the name. That's why I added a bug report to my posting. ;)
    phpBB: The long, sad history of security issues, combined with the recent(ish) approach to "just throw a framework (Symfony) onto it" to circumvent them.
    • Informative! Informative! x 1
    • List
  19. haqzore

    haqzore Devotee

    Appreciate the context, but seems a bit light on substance to warrant a cynical approach to all software.

    Granted, this is only context relating to 2 specific scripts, so maybe your past experiences do warrant it.
  20. \o/

    \o/ an oddity

    They do. I need to give AsmBB a second look ... ;)
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.