I'm opening a video upload service in my Forum and i have to transfer (or open) a new storage in AWS

Discussion in 'XenForo' started by palhanow, Oct 3, 2019.

  1. palhanow

    palhanow Enthusiast

    122
    105
    +114
    Hello guys,

    So, little by little my Forum is growing with my site. And in the process to expand my base of operations, I'm creating a video upload service with the XFMG.

    Is not a directly competition with YouTube but is a safe space for people willing to upload their videos without the fear of been deleted or get banned for some 'policies' about this network.

    Since my actual host don't provide a good price for storage purposes, i have to contract the Amazon Web Services or Digital Ocean Spaces to have my videos stored there.

    XenForo actually provides a nice add on for this kind of stuff by a Member of the Staff:

    https://xenforo.com/community/resou...or-amazon-s3-for-file-storage-in-xf-2-x.6805/

    So, I'm here to discuss with you guys all the aspects of make this thing happen. I'm not entering in the monetization subject or if is going to be a success.

    I'm here to ask the users in this Community about the technical things or even, contract you, experienced and cheaper developer for do this to me. In the XF Official Forum, I'm having offers around 75 to 140 U$ for make this happen.

    So, what i need basically is to have a good storage server, like Digital Ocean Spaces or Amazon Web Services, make the changes in the config.php file and install the add on, right?

    After this, just install the Media Manager and open for the public.

    One point that I'm having some questions is, what server is gonna be responsible for the playback. Storage is ok, is going to happen, but and the transmission of the video? Happens in the Amazon/DOS side or in the original side?

    I have this question because my original server, where my website are, is very simple and i think they can't run a decent video platform without buying more upgrades for the server.

    So, I'm opened for debates guys. Please, light me.
     
  2. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111
    I'm not sure I'd be looking for external storage for video hosting personally;

    • Amazon S3's charging structure is based on I/O usage. For me this would rule out video usage.
    • DO spaces are 250GB and i'm not sure this can be scaled up. You would max this out for video usage quite quickly.
    • Although I think both have their own CDNs, i'm not sure its a great idea performance wise.
    You say the original server is "simple" - shared hosting?

    If you want to seriously run a video platform, you may need to consider a dedicated platform.
     
    • Informative! Informative! x 1
    • List
  3. palhanow

    palhanow Enthusiast

    122
    105
    +114
    Thanks for your replies.

    Yeah, a flexible server with demand would be great for videos and stream demand and this make me go to my main question.

    My original server is was way pass the shared hosting level and if a look the first server that i rend, man, i want to cry like a kid who lost his candy. Actually is not a dedicated, but my VPS/Cloud works fine with my actual demand, some bumps here and there, but per U$ 50 bucks a month i cannot complain.

    My main concern is, what server is gonna be responsible for all the "stream"? The video playback, per say.

    My idea is to keep the site running in my original server and the playback, all by Amazon, since DO spaces are very limited, like you said.

    I'm asking because i don't know if my premise is correct.

    Since Amazon is gonna be all about the storage of the video files, i think they gonna play the videos too, right? And my original server, well, just open the XF platform.
     
  4. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111
    No, AWS won't be playing it. Their CDN will keep it stored and cached across their networks which will reduce your bandwidth and increase speed, but the encoding im sure is done at base level, your server.
     
  5. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111
    I'd have thought you would need a fairly significant amount of RAM for it too
     
  6. Threadloom

    Threadloom Sponsor

    107
    55
    +89
    How big will these files be? If you are talking 100M or larger for more than a 30-second mid-quality video you are going to quickly run out of processing and space. Using S3 will increase your costs, considerably for transferring in and out.

    Could you set up a system where it uploads the video to youtube/vimeo to your own channel and posts from there back to your forum? That way storage, and transfer if all on youtube/vimeo...plus you'll be gaining content, etc? I would recommend looking at any legal ramifications, along with updating TOS, etc.
     
  7. Joel R

    Joel R Fan

    710
    257
    +757
    I know you posted this in Xenforo, but I've explored something similar for my Invision Community installation.

    To make it clear:
    1. Your server will handle any transcoding.
    2. Amazon S3 will handle the storage and initial delivery.
    3. Amazon Cloudfront or your CDN will handle the final end-user delivery.

    Some thoughts:
    - I don't know how deeply you've explored this option, but I've personally run away from video streaming. It is extremely hard to do well, to the expectations that modern consumers want. Users want multiple bit rate transcoding for crossbrowser and crossdevice compatibility, they want automated thumbnail generation, they want instant and auto-bitrate streaming, they want language subtitles, they want a custom video player to skip forward or slow down, and they want it all for free.
    - Amazon S3 is cheap for storage. Amazon S3 or Amazon Cloudfront is expensive on the bandwidth. Even if you use a free CDN like Cloudflare, your videos will need to first transfer from Amazon S3 to Cloudflare, so you will still incur egress bandwidth charges. I run a media-driven website, and only 10% of my Amazon AWS bill is related to storage. The other 90% are all bandwidth fees.
    - Video, and especially short form video on mobile, is the future of communication yet all of these legacy forum platforms are so far behind the curve. They incorporate video in the most superficial of manners.
     
    • Informative! Informative! x 2
    • List
  8. LeadCrow

    LeadCrow Apocalypse Admin

    6,441
    1,232
    +2,181
    If you'd just prefer a cost-effective solution that includes transcoding and streaming for a large number of short videos, Cloudflare's Stream might be worth consideration.

    I'd however suggest evaluating Peertube if you're concerned about costs and need some control. Its peer2peer streaming system drastically reduces the original server's bandwith consumption and should leverage you current server's spare capacity well.
    No integration with xenforo exists currently, but it works better as its own website open to non-members of the forum anyway and you can embed p2p-enabled streams into the forum or gallery just as seamlessly as you can with youtube videos, with the benefit of bandwidth saving or extra privacy if you select to only load videos from the server copy (as on classic media galleries). Either approach should work whatever storage provider you go with, and transcoding shouldnt be necessary for imports from other streaming services.
     
  9. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111
    Just to extend off my original comment. DO do allow you to extend that 250GB limit, its chargeable at $0.02 per GB. So if you extended this to 1TB you'd be looking $20 a month.
    The bandwidth, which to me is the crucial part here, is also chargeable after 1TB at half that amount.

    TBH, once you start going into the territory of extending that space, you may as well be looking at a dedicated server. If not for the pricing POV the security side of it. If something goes wrong I wouldnt be comfortable having that amount of data external to my main machine. I have heard of people having billing issues with DO and them revoking access. Not what you want with a terabyte of data that is paramount to your forum.
     
    • Informative! Informative! x 1
    • List
  10. pierce

    pierce Habitué

    1,169
    262
    +720
    Look at back blaze. They do B2 integrations with cdn's and server providers for free transfer. Can remove you completely from the loop!

    Streaming in this page:

    https://www.backblaze.com/blog/b2-cloud-storage-roundup/

    There's no super low cost method and you should charge a monthly for hosting video!
     
  11. Paul M

    Paul M Limeade Addict

    3,819
    1,627
    +2,235
    Unless you are talking small files, and small usage, I would not bother.
    Amazon S3 will very quickly start to cost you a fortune, in storage and bandwidth.
    DO is unlimited storage, but charges for every GB over 250, plus any BW over 1TB .

    Transcoding multiple files will likely kill any shared server, and probably most VPS's.
    I run a dedicated video procssing server, and its reasonably powerful, but I still have to limit what it can do simultaneously.
     
  12. LeadCrow

    LeadCrow Apocalypse Admin

    6,441
    1,232
    +2,181
    At some point, a physical server handling storage, transcoding and service would make the most sense, but upload bandwidth and speed would remain troublesome since it's usually limited or charged for beyond a certain threshold. Selfhosting bandwidth-saving solutions like peertube drastically reduces the associated costs and the risk of those affecting your other project's funding.
     
  13. MagicalAzareal

    MagicalAzareal Magical Developer

    435
    332
    +206
    If you want to do things like video transcoding, you might need some sort of GPU to do it most efficiently, although that is an upfront cost of it's own, if you don't have enough volume / content to feed it.
     
  14. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111
    This quote interests me too;

    Youtube has "such" policies for a reason, to keep its freedom in ensuring its platform is safe from illegal, inappropriate or dangerous material.

    This may be different for you at the moment as your member base is likely smaller, and as such more trusted. If this grows to more unknown audiences those differences may disappear.

    I would think very carefully about basing your service on such different policies, you may even find that whatever infrastructure you implement may require you to do so.
     
  15. MagicalAzareal

    MagicalAzareal Magical Developer

    435
    332
    +206
    I can understand deleting the ElsaGate teaching kids to kill themselves disturbing horror show, I don't even know what would possess someone to produce that, but they're going after a lot, including even UFO conspiracy videos and ones saying the world is flat. Those are more entertainment than anything someone would actually take seriously.

    A lot of these tech giants have kind of just lost it, although the algorithm was never that great.
     
  16. Dubbed Navigator

    Dubbed Navigator Adherent

    302
    105
    +111

    I remember that one, was keeping an eye out so my daughter didnt watch it. Some sick puppies out there.

    The point wasn't their moderation policy, more restricting yourself out of the freedom of having one. If you start the service based on saying to people we wont moderate or delete your videos, you face more of a struggle to implement it further down the line.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.