Determining Your "BEST" Members WITH Data!

Discussion in 'Members & Staff' started by ProWeb, Mar 11, 2017.

  1. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    One of the most important tasks for a forum management team (Admins, Moderators, etc.) is to find and reward / encourage the "best" members. Now, keep in mind this term is VERY subjective and I'm not talking about the individual's value in REAL LIFE... I'm saying from the perspective of "contributions to the health and growth of a community" perspective.

    I think I'll eventually even build out an xf addon that displays this info for moderators and allows for fine-tuning the calculations.

    Right now, these are the main data points I'm using to find the "best" members:
    • Total Posts
    • Posts Last 60 Days
    • Total Likes Received
    • Likes Received Last 60 Days
    I calculate a "total score" based on a weight I give each of these (like total posts/70 + posts-60-days/4 + total likes/3 + Likes-60-days/2)... so, I get a combo of the "best" members from all-time and recent. I can tweak the weights to call out just the all-time or just the recent / up-n-commers.

    I could throw in all kinds of other data points like:
    • Threads started
    • Images uploaded
    • Articles created
    • etc.
    I could also fine-tune how the data is pulled, for example:
    • Instead of just using "all likes", I could segment it so it's only likes from non-social sections of the forum. I could also give more weight to likes give to articles vs. posts.
    • I could give more weight to posts in the "on-topic" sections and decrease the weight for "social" sections.
    This, of course, brings up the discussion of value of social engagement vs. topic-specific engagement. Does it matter what the content is, as long as there's a lot and other members are liking it?

    So, what data points would you use to tease-out your "best" members?
     
  2. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    I'm digging up this old thread because I'd LOVE some help in how to really tease out (with data) who the "best" members are.

    Obviously the easy one is a combo of # of posts and # of likes... and then post-to-like ratios.

    ... but I'd like to get way more nuanced / detailed than that.

    What other ways can we use the tons of backend data in our DB's to tease out best-members?
     
  3. PoetJC

    PoetJC ♠ Jacquii: Black Kween of Hearts ♠

    21,066
    1,497
    +5,109
    A couple suggestions:
    1. Specifically communicate with said member(s) via PM, thanking them for loyalty personally so that they understand their value to the community.
    2. "Best" members (using your terminology) need to see an upgrade in their membership status. Perhaps offer them a moderator position...
    "Teasing" is not gonna work as for inspiring participation. What you need to do is continually provide valuable content so that members will feel the need to join the conversation. And for the love of God.. Create a posting policy which encourages conversation, while encouraging your moderating team to "not be scared" about joining in, even when the topic(s) are contentious ones. What you don't do is create an atmosphere where members feel tentative about voicing their various opinions; we aim to keep the conversation on topic while encouraging honest feedback from our users.

    Intriguing topic ProWebProWeb ... Wish I could add more and probably will.
    One point of contention ==> "Content is King" .... Using such logic would dictate that "Users (Members) are not King."
    Lay out a plate of cake and folks will want the corner pieces.. The corner pieces have more icing and we have a sweet tooth. You want to encourage your users to be comfortable in exposing their sweet tooth in such a manner that demands ice-cream be added to the plate. Even we without the corner cake pieces can feel included and perhaps even ask for a double-scoop, while (staff member or not) continually adding substantive content to your database.

    We won't talk about the chips..

    Bad analogy I'm sure LOL - But it's all that I have to offer at this particular moment in time.
    Cheers dude.

    J.
     
    Last edited: Jun 9, 2018
  4. jair101

    jair101 Aspirant

    28
    3
    +11
    I am actually thinking about something similar, but I want to find a way to create a rating for each post:

    - based on in which forum it is posted - posts in chit-chat forums/topics should count less
    - based on size of post - one liners should count less then long thoughtful answers
    - based on amount of reactions. I even think of introducing some reactions that will be available only for mod/admin team and they will add additional boost
    - based on number of attachments - in my niche (travel), it is nice to post a lot of images, so this should be encouraged.
    - based on additional articles, gallery albums, etc - not just the forum content

    You can also create some per member factors:

    - how often they log in
    - last time logged in (I guess points should be decaying if they don't login too much, I don't want members that haven't logged in in 2 years carry a lot of rating)
    - etc...

    You can make more complicated ones as well:

    - members who don't react to posts a lot, should have their reactions count more then those that like everything
    - do the members provide feedback after they ask question and receive answers, this should give more points
    - etc.

    Then simply sum all points per user and you get the most useful ones.

    I haven't been able to formalize my entire idea yet, but it will be something similar to this. I plan to do it in the next few months, I hope I won't need any custom work done, but it is very improbable I will be able to escape that. Currently using IPS and planning to do it with the automation rules addon.

    It is a real shame that currently we are only able to rank members based on very artificial factors like post, topic and reputation counts.
     
    • Pure Genius! Pure Genius! x 1
    • List
  5. Wes of StarArmy

    Wes of StarArmy Adherent

    323
    105
    +115
    I agree that post size should be a factor.

    One issue is that finding the most helpful members is only one side of the coin - having that data but only the staff seeing it seems like a missed opportunity. If you could tie it to a rep system or point system or the Xenforo trophy system, where you can gamify activity, incentivize activity, and make it competitive, that would be a real winner. Some of the old cash mods and such did something like this (Like where Gaia Online rewards "gold" for posting) but not as advanced.

    Also should reports or warnings against the member be factored into your scoring system?
     
  6. Alfa1

    Alfa1 Moderator

    3,467
    1,202
    +2,403
    For something like this I think you need to first measure post quality. And for that likes are not enough, because you also need to measure bad posts and how bad.
    If a reputation system is integrated well then members will rate rule breaches negatively. Example from my reputation system of a troll getting rated:

    Screenshot-2018-6-14 What happened i'm confused .png
    Hence warnings are effectively factored in, in the reputation of a member.
    Suffice to say that the above member was warned & banned.

    If low reputation thresholds affect usergroup demotion then this boils down to crowd moderation.
    Conversely if high reputation thresholds are a factor in promotions, then members with stellar contributions will quickly rise in ranks and become visible to staff and members.

    Another factor we use is post to reputation factor.
     
  7. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    Brilliant discussion, thanks all!!

    YES!!! We definitely would want to weigh the posts (and corresponding likes) of the real content areas of the site vs. chit-chat / random-game content. Obviously any type of engagement has value, so I don't want to completely disregard non-topic posts... but I would also like to weight them differently.

    This seems like it would be possible, but I bet the query time (to run on each post) on a huge forum would be catastrophic. Maybe limiting it to just content posted in the last 30 days would be a good proxy?

    In the tests I've been doing, a lot of math needs to go into this to bring the numbers, and how the interact with each-other into the mix. I do a lot of multiplication and division to equalize things. I bet there's a more scientific way to do this... for example, averaging all the data across all users, then determining +/- % above and below the average. That's probably the best way to equalize everything from "how many year's they have been an active member" (using double-digit or less) vs. how many likes they have received (often in the thousands)

    Exactly! We want to use the info for our moderation team to show appreciation, but also so we can award, gift, etc. these members!

    My gut says no. We already have a problem where our most active members also tend to get the most warnings/infractions... just simply by the fact that they are so active. I guess this could be a factor if we deploy it in a way I mentioned above.. % above / below the average. So if most members have 0.01% post-to-warning ratio, and a member has a ratio of 1%, that may indicate a problem. Actually... this might be a GREAT way to quickly tease out how relatively "good" / "bad" a member is. I'm gonna need to play with this a bit (NOTE TO SELF: infraction to post count ratio!!) ;)
     
  8. we_are_borg

    we_are_borg Moderator

    4,395
    807
    +1,691
    If i remember correctly DigitalpointDigitalpoint had a very good system in place for posts but it was more then reputation.
     
  9. Nev_Dull

    Nev_Dull Anachronism

    1,714
    717
    +779
    It's an interesting idea. I have questions, though.
    • How do you measure quality? Quality is very subjective, depending more on the reader than the writer.
    • How do you measure relevance? Someone can spend a great deal of time and effort in creating a post and completely miss the point.
    • Assuming you can quantify all of this, what do you do with the data? In my experience, those posters who regularly provide useful, high-quality content do so because they care about the topic and the community. They will continue to post regardless of recognition or reward.
     
  10. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    Nev_DullNev_Dull you're spot-on with your questions... and hence the existence of this thread.

    I doubt there is going to be a perfect answer to any of your questions... even with unlimited data. As you say, it's too subjective to have a perfect qualitative measure.

    That said, I think there might be ways to use the data we DO have to start isolating metrics to help act as a proxy for "best" content, contributions, and members.

    Right, so I'm wondering if using a combination of likes, views, posts, quotes, acceleration/velocity of likes/views/quotes, etc. will help identify which content the MOST readers think is most "valuable".

    I don't think there's a good way to do this, but if I step back, I actually don't think it's that important. I mean, if someone posts content to my forum that is completely "irrelevant" but TONS of members like it, quote it, reply to it, share it, etc., then that's a win for the community (and the site). I think my main goal is to really find the most "valuable" content which can then be used as an additional metric to help add to finding the most "valuable" members.

    ... while writing that, I started thinking about the various types of members. Some that are "valuable" because they are extremely friendly, welcoming new members, liking people's content, etc., but who may never really post topic-related content... or may not even post a lot of content, but when they do, it's quality over quantity.

    Great question, and hopefully it will be a mix of things. I do disagree with your point that they will "continue to post regardless of recognition or reward." Some might, but most people are motivated by some very specific "rewards". I think the best way for me to answer that question is to refer to one of my favorite videos on what motivates people:

     
  11. Nev_Dull

    Nev_Dull Anachronism

    1,714
    717
    +779
    Could those not also denote the most controversial or contentious as well? That may not matter, depending on how you define 'valuable' content.


    Very good point. Not all important contributions are the same. It also depends greatly on the type of forum. The most valuable posters on a more technical forum might be the ones that provide simple, definitive answers to questions. They may rarely get quoted, liked or otherwise noted because people don't always respond after they have the answer they need.

    While I'm all for showing appreciation to those who provide value for the community, it seems like a lot of work just for that. Generally, those posters are well-known and sadly, in a distinct minority.

    Lots of things to consider. I don't envy you trying to put it all together.
     
  12. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    Great point, but fortunately (at least on all my communities) this is VERY self-policing. Any time a discussion's tons of views, replies, likes, etc. are driven by contention / controversy, it usually degrades so precipitously that people quickly break rules (troll / flame) and then the topic is shut-down and/or people are banned. So, at least in our cases, we'd never see a connection between "best" content or "best members" being driven by contention or controversy.

    Yes, definitely a lot of work discovering, recognizing, and rewarding the "best" members, but like anything that involves human behavior, work, contribution, etc., keeping them happy and feeling appreciated (whether it be an employee, a customer, a community, a spouse, etc.) has the highest potential ROI when compared to most other efforts.
     
  13. ProWeb

    ProWeb Enthusiast

    196
    83
    +23
    Oh, one more point related to this:

    We're trying to use the data from our LONG-TERM "best" members and apply that data to identify (as a proxy) the "up-and-coming" potential best members. If we can identify and encourage them early on, we hope that will increase the probability of them growing into future long-term best members.

    Kinda like if you're given 100,000 seeds, and can only plant / nourish 50 of them, it would be awesome if you could use some data to help identify which ones have the highest potential to turn into huge-strong productive plants vs. the ones that don't sprout or die after a few months regardless of how much effort you put into them.
     
  14. Joeychgo

    Joeychgo TAZ Administrator

    6,653
    1,532
    +3,348
    average length of posts
    average likes per post
     
  15. diadi

    diadi Enthusiast

    134
    43
    +77
    A sentiment analysis could help with this.

    https://cloud.google.com/natural-language/docs/sentiment-tutorial
     

    Attached Files:

  16. Joel R

    Joel R Fan

    511
    187
    +482
    This is an intriguing post. I didn't have time to read everything so far, but I get your question: how do you identify valuable members?

    I know you're developing for XF so I apologize in advance if I bring in some IPS concepts.

    In IPS, we have the concept of a leaderboard. It shows daily rankings of 1sr, 2nd, and 3rd place based upon reputation. Reputation is determined by the aggregate number of reactions. It also shows the content items with the most reactions. This can be adjusted on a daily, weekly, monthly, etc basis to see the changing leaderboard.

    In terms of macro benchmarks, these are the two meadures: quantity and quality. However, I think you can then get to a micro-level and begin to dissect ...

    Quantity - is 100 image uploads in one album the equivalent to 100 unique, handwrittenhblog posts? Is 100 forum posts in a social, general chit chat topic the equivalent to 100 incisive responses in a topic of hard knowledge? Is getting a best answer in a Q&A equivalent to posting the question? All of these require more fine grained control over what's most important to the admin in terms of measuring quantity.

    Quality - You normally measure this by reactions. But is getting 100 funny reactions the same as 100 informative reactions? You can also measure quality by posts that are either manually or system-picked for promotion, feature, etc. For example, in IPS we have a small feature called "popular." A content item that gets a certain number of reactions in a certain time gets a popular flag, and they show up in a special feed. That could be a way of measuring quality authors if they continue to write posts that achieve popular status.

    There's also a network effect. Certain users may not write a lot, but when they do, they're heavily referenced or quoted. They may also have a lot of followers. That could be a measure of their social impact among their peer group.

    In any case, it would be lovely to see this project of yours progress. It definitely sounds like a rather interesting social experiment.
     
  17. Pete

    Pete Flavours of Forums Forever

    1,773
    227
    +599
    Would you factor in time for the users too? For example, a member that's been around for a year getting a total of 12000 reactions (i.e. 1k per month avg) vs a member that's been around a month and achieved 2000 reactions?
     
  18. Joel R

    Joel R Fan

    511
    187
    +482
    Yeah that's a great point and goes to the popular feature that I was talking about, where it's measured on a rolling basis within a definitive time frame.
     
Verification:
Draft saved Draft deleted
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.