Gosora - Supremely fast forum software

Discussion in 'Forum Software Development' started by MagicalAzareal, Apr 30, 2019.

  1. MagicalAzareal

    MagicalAzareal Magical Developer

    195
    48
    +74
    I'm back here after I dunno four years, I can't remember, my perception of time is too awful. See my introduction thread for more details on who I am.
    I've been experimenting with Go, big thanks to Euan for pointing that language out and I threw together a bug tracker which morphed into a custom forum software.

    https://gosora-project.com/

    https://github.com/Azareal/Gosora/blob/master/README.md

    I was inspired to create it by the RR scaling incident, a lack of good systems which met my needs, a lack of systems which could keep pace with the ever changing world and, most importantly:
    To learn a new programming language and to experiment with more low level technologies in a web context (I'm slowly digesting the knowledge I got from reading the HTTP stack implementation Google wrote for Go).

    It has a bunch of features which are essential to me, but probably more boring for others.

    * Live dashboard showing me how much CPU, RAM, etc. is being used on the system. I love this feature.

    * Live Notifications when someone does something, because that's kind of important to me. Also, some things like the topic list are live, but I never quite got around to rolling that everywhere.

    * I try to keep the stack simple in true Go style. There is no requirement to have Node installed or Redis or Imagick or any manner of things which complicates operations. Cron jobs or other task schedulers aren't required, as the main process handles tasks and what-not itself.

    * Zero cost analytics (I cheat by holding the data in memory and batching them to the database every now and then).

    * Zero queries on hotpaths (pull random things out of memory).

    * Asynchronous programming without any of the pain (except for data races, I hate those x.x).

    * C / Rust level performance from Go absent GC woes and the standard library over-allocating (Go 2 is the magic word usually used for problems like that).

    * Experimental transpilation to client JS and server-side Go. This one was partly for fun, but it works well too.

    * I avoid C and Assembly for security reasons where possible. I might not be able to sleep from the ensuing paranoia, I'll see if I can introduce some more when I add support for Docker and am able to safely isolate those elements away. Maybe process level isolation might be enough?

    * I leave out a fair bit of fluff which I think distracts from the core experience.

    * Can probably be run off a toaster. My laptop with constantly maxed out memory usage from Firefox, Opera and Chrome waging war on each other is probably a good example, but there are also smart toasters out there these days.

    * Four default themes. You can switch between them via the theme selector in the footer. Nox is the defaultest default and Cosora is the most advanced light one.

    * More in https://gosora-project.com/topic/features-introduction.72

    The analytics includes route counters, UA agent counters, referrer counters, memory usage tracker (coming soon), etc. and has graphs plotting out this data over time.

    I take care to anonymise all of the data which I collect by aggreggating all of the data together to make sure that the privacy of my users is adequately preserved, not just because "GDPR", but because privacy is a fundamental human right.

    I plan to expand on that by periodically purging request logs (standalone), IPs associated to posts / users, etc. But that'll probably be a setting of some sort.

    Very crude benchmarks show throughput over a thousand times faster than MyBB which on it's site claims to be "one of the fastest forum software", although it wasn't anything particularly scientific and will likely vary depending on the environment.

    MyBB was picked as my guinea pig as it was always my favourite software (even if I've outgrown it now) and it wasn't as difficult to setup for tests as phpBB. I also tried NodeBB, but it short-circuited on a too many requests error page, so I couldn't get it's true throughput, but the error page wasn't too performant.

    It can be run as a standalone server or proxied behind Nginx or whatever fits one's needs.

    I do have to note that I might be adding support for HTTP/3 soon which may not be compatible with Nginx and the like, but which does improve client performance by a fair bit, so things might get complicated there (or just turn that off).

    Future stuff:

    * Finish adding ElasticSearch, or find an equivalent library for Go, to reduce the amount of ops work I have to do.

    * Finish hardening it. I am paranoid about security, so this is a task which basically never ends.

    * Implement "social" stuff like friending and maybe customising profiles?

    * Support databases other than MySQL and the partly broken MSSQL.

    * Add a more sophisticated updater than the one I have now.

    * Figure out how we can beat AsmBB, if we don't already lol

    * Move gosora-project.com to a better host.

    And before you ask. It does not run on shared hosts.
    Well... Someone managed to get it to work on one, but I don't advise it.
     
    Last edited: Apr 30, 2019
  2. MagicalAzareal

    MagicalAzareal Magical Developer

    195
    48
    +74
    I forgot to mention something very important. There are four default themes.
    You can switch between them via the theme selector in the footer, so if you don't like dark themes (Nox is the defaultest default), then you might want to take a look at Cosora :)
     
  3. MagicalAzareal

    MagicalAzareal Magical Developer

    195
    48
    +74
    https://gosora-project.com/topic/patches-18-to-20.105
    Some things I've done since April 8th.

    I'd like to eliminate server rendering for users with JavaScript enabled entirely (this'll be a fun one, it's easy, but making it performant will be the real challenge as always lol), periodic IP purging and maybe HTTP/3.

    HTTP/3 is slightly annoying though as it involves opening an extra port, which requires an extra listener and it's incompatible with just about every CDN making it useless on any serious site. Even Cloudflare, the most advanced CDN, has yet to roll out support for HTTP/3.

    CDNs are very very important, much more so than HTTP/3.
    I've seen users on some forums suffer from eight second load times when it could have been a fraction of that, because the admins were too stubborn to use a CDN.
     
  4. cornnfedd

    cornnfedd Captain Futurama

    906
    547
    +205
    Wow that is super fast!!
     
  5. MagicalAzareal

    MagicalAzareal Magical Developer

    195
    48
    +74
    Thanks, I've been doing loads of client optimisations to get things faster on mobile and particularly in places like Australia where pages can take an annoyingly long time to load.
     
  6. MagicalAzareal

    MagicalAzareal Magical Developer

    195
    48
    +74
    I've done a few things, although I've been a little depressed (alright, not a little), so a bit less than I would have liked.

    Some of the things I say might go over your head, if you don't have a certain amount of knowledge about computer science, so I'll briefly explain some of the basics.
    Just stuff you'll get from skimming any random C++ tutorial. I might skim over some of the finer details, so please don't bite me lol

    Computer memory is a giant grid of sorts. Each slot of sorts has it's own memory address.
    Data is stored in these slots and you can access these bits of data via simply named things known as variables.

    There is one type of variable however called a pointer which instead of holding a bit of data in that memory slot will instead hold a number pointing to another memory slot.
    Now, you might be wondering. Why would you do that?

    Well, instead of making copies of the same thing over and over in a number of slots, you can just point a bunch to the same slot and reuse the contents in lots of different places and repoint elsewhere when you want to change whatever is backing it.

    Basics taken care of. Onto the actual changes.

    I've added a simple spam filter to deal with the spammy referrer bots, as they were becoming a bit of an eyesore. You can still click on the filter to expose them.
    This one was snatched from a lovely dictionary which I'm very fond of, but you can now do site searches from the URL Bar.

    https://gosora-project.com/topic/memory-usage-chart.104
    I added a couple of analytics panes to plot the amount of memory Gosora has used over time on a nice little chart. Go tends to request more memory than it needs from the OS, so there's a chart showing actual memory use and the amount of memory that it has reserved.

    https://gosora-project.com/topic/debug-page.107
    I also improved the UI for the debug page, it's largely for debugging the program, so it may be hard to follow. And yes, I do need to upgrade the database on that specific local build.

    I'm also experimenting with a small memory reply cache. Not much to see here, as I'm still collecting data to see how well it does.

    I've implemented meta store which is a small key-value store, just for simplicity in a few areas, like handling daily tasks.

    Topic / Reply / Profile Reply IP Addresses are now purged from the system after 180 days elapse for privacy reasons. You can configure or disable that in config.json, there is more to come here.

    I've added a couple of provisional exclusion flags to the analytics panes. You currently need to know the right URL parameter to make use of this. This was mainly added so I could reduce the amount of noise on the routes / languages analytics panes, as one particular route / language would tend to dominate the charts.

    I have also made some efforts to dramatically reduce the amount of bandwidth being used in the steady state.

    I've also fixed a number of bugs including one where the WOL widget would wake up too frequently to rebuild itself and trigger associated plugin hooks even when nothing had changed.

    I'm doing a rewrite of the hashlink, mention and URL parser right now.

    So far, I've dramatically reduced the number of edge cases (over 70 new test cases), improved performance by reducing the number of memory allocations and reduced the data moving between the CPU and main RAM by using the same pointer for Content and ContentHtml when the two are identical.
     
    • Like Like x 1
    • Informative! Informative! x 1
    • List
Verification:
Draft saved Draft deleted
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.