Skip to content
linux-nerds.org

linux-nerds.org
  1. Übersicht
  2. ActivityPub Test Kategorie
  3. Increased memory consumption after upgrading from v3.7.x to v4.2

Increased memory consumption after upgrading from v3.7.x to v4.2

ActivityPub Test Kategorie
  • L

    Hey, so after the upgrade the memory consumption has been creeping up at a fairly steady rate and exceeding 1GB of memory. Is this something to be expected from the version upgrade? Or could it be due to our own changes / plugins we have? Or due to scrapers perhaps?

    Upgrade happened on Monday 14th and before that the memory consumption was fairly steady at around 500MB. After the upgrade the memory consumption has been steadily going up during the day as seen in the pictures:

    Screenshot when the sudden increase was noticed:
    9119758a-4530-497e-a570-8f11d5d17015-image.png

    Here is last 7 days: (the falling edges are due to daily restart)
    a180780e-92ff-437f-bda7-b50c5265704b-image.png

    Screenshot from the /admin/development/info on the 23rd when it started also using swap.
    bdf69f0e-459b-4239-96f6-d8b89b7e7834-image.png

    Active plugins:
    * nodebb-plugin-s3-uploads@3.0.5 (installed, enabled)
    * nodebb-plugin-2factor@7.5.9 (installed, disabled)
    * nodebb-plugin-composer-default@10.2.47 (installed, enabled)
    * nodebb-plugin-dbsearch@6.2.13 (installed, enabled)
    * nodebb-plugin-emoji@6.0.2 (installed, disabled)
    * nodebb-plugin-emoji-android@4.1.1 (installed, disabled)
    * nodebb-plugin-gravatar@4.0.0 (installed, enabled)
    * nodebb-plugin-markdown@13.1.1 (installed, enabled)
    * nodebb-plugin-mentions@4.7.1 (installed, enabled)
    * nodebb-plugin-poll@3.2.5 (installed, disabled)
    * nodebb-plugin-question-and-answer@1.2.7 (installed, enabled)
    * nodebb-plugin-spam-be-gone@2.3.1 (installed, enabled)
    * nodebb-plugin-sso-custom@1.0.0 (installed, enabled)
    * nodebb-plugin-web-push@0.7.3 (installed, disabled)
    * nodebb-plugin-write-api-custom@5.8.12 (installed, enabled)
    * nodebb-rewards-essentials@1.0.1 (installed, enabled)
    * nodebb-theme-harmony@2.0.40 (installed, disabled)
    * nodebb-theme-lavender@7.1.18 (installed, disabled)
    * nodebb-theme-peace@2.2.39 (installed, disabled)
    * nodebb-theme-persona@14.0.16 (installed, disabled)
    * nodebb-theme-custom@1.0.0 (installed, enabled)
    * nodebb-widget-essentials@7.0.36 (installed, enabled)

    Nginx is being used to serve static assets and the config is the same as it was before the upgrade.

    Setup:
    Mongodb Atlas M40
    Redis is Heroku's own key value store premium 2
    Heroku -> 4 dynos (Standard-2X variant)
    In one dyno we have nginx + nodebb

    Any help would be greatly appreciated!
    Thanks in advance.

    baris@community.nodebb.orgB L julian@community.nodebb.orgJ 3 Antworten
    0
  • L lassesuom@community.nodebb.org

    Hey, so after the upgrade the memory consumption has been creeping up at a fairly steady rate and exceeding 1GB of memory. Is this something to be expected from the version upgrade? Or could it be due to our own changes / plugins we have? Or due to scrapers perhaps?

    Upgrade happened on Monday 14th and before that the memory consumption was fairly steady at around 500MB. After the upgrade the memory consumption has been steadily going up during the day as seen in the pictures:

    Screenshot when the sudden increase was noticed:
    9119758a-4530-497e-a570-8f11d5d17015-image.png

    Here is last 7 days: (the falling edges are due to daily restart)
    a180780e-92ff-437f-bda7-b50c5265704b-image.png

    Screenshot from the /admin/development/info on the 23rd when it started also using swap.
    bdf69f0e-459b-4239-96f6-d8b89b7e7834-image.png

    Active plugins:
    * nodebb-plugin-s3-uploads@3.0.5 (installed, enabled)
    * nodebb-plugin-2factor@7.5.9 (installed, disabled)
    * nodebb-plugin-composer-default@10.2.47 (installed, enabled)
    * nodebb-plugin-dbsearch@6.2.13 (installed, enabled)
    * nodebb-plugin-emoji@6.0.2 (installed, disabled)
    * nodebb-plugin-emoji-android@4.1.1 (installed, disabled)
    * nodebb-plugin-gravatar@4.0.0 (installed, enabled)
    * nodebb-plugin-markdown@13.1.1 (installed, enabled)
    * nodebb-plugin-mentions@4.7.1 (installed, enabled)
    * nodebb-plugin-poll@3.2.5 (installed, disabled)
    * nodebb-plugin-question-and-answer@1.2.7 (installed, enabled)
    * nodebb-plugin-spam-be-gone@2.3.1 (installed, enabled)
    * nodebb-plugin-sso-custom@1.0.0 (installed, enabled)
    * nodebb-plugin-web-push@0.7.3 (installed, disabled)
    * nodebb-plugin-write-api-custom@5.8.12 (installed, enabled)
    * nodebb-rewards-essentials@1.0.1 (installed, enabled)
    * nodebb-theme-harmony@2.0.40 (installed, disabled)
    * nodebb-theme-lavender@7.1.18 (installed, disabled)
    * nodebb-theme-peace@2.2.39 (installed, disabled)
    * nodebb-theme-persona@14.0.16 (installed, disabled)
    * nodebb-theme-custom@1.0.0 (installed, enabled)
    * nodebb-widget-essentials@7.0.36 (installed, enabled)

    Nginx is being used to serve static assets and the config is the same as it was before the upgrade.

    Setup:
    Mongodb Atlas M40
    Redis is Heroku's own key value store premium 2
    Heroku -> 4 dynos (Standard-2X variant)
    In one dyno we have nginx + nodebb

    Any help would be greatly appreciated!
    Thanks in advance.

    baris@community.nodebb.orgB

    I would disable the daily restarts to see if the memory usage keeps going up or stabilizes.

    0
  • L lassesuom@community.nodebb.org

    Hey, so after the upgrade the memory consumption has been creeping up at a fairly steady rate and exceeding 1GB of memory. Is this something to be expected from the version upgrade? Or could it be due to our own changes / plugins we have? Or due to scrapers perhaps?

    Upgrade happened on Monday 14th and before that the memory consumption was fairly steady at around 500MB. After the upgrade the memory consumption has been steadily going up during the day as seen in the pictures:

    Screenshot when the sudden increase was noticed:
    9119758a-4530-497e-a570-8f11d5d17015-image.png

    Here is last 7 days: (the falling edges are due to daily restart)
    a180780e-92ff-437f-bda7-b50c5265704b-image.png

    Screenshot from the /admin/development/info on the 23rd when it started also using swap.
    bdf69f0e-459b-4239-96f6-d8b89b7e7834-image.png

    Active plugins:
    * nodebb-plugin-s3-uploads@3.0.5 (installed, enabled)
    * nodebb-plugin-2factor@7.5.9 (installed, disabled)
    * nodebb-plugin-composer-default@10.2.47 (installed, enabled)
    * nodebb-plugin-dbsearch@6.2.13 (installed, enabled)
    * nodebb-plugin-emoji@6.0.2 (installed, disabled)
    * nodebb-plugin-emoji-android@4.1.1 (installed, disabled)
    * nodebb-plugin-gravatar@4.0.0 (installed, enabled)
    * nodebb-plugin-markdown@13.1.1 (installed, enabled)
    * nodebb-plugin-mentions@4.7.1 (installed, enabled)
    * nodebb-plugin-poll@3.2.5 (installed, disabled)
    * nodebb-plugin-question-and-answer@1.2.7 (installed, enabled)
    * nodebb-plugin-spam-be-gone@2.3.1 (installed, enabled)
    * nodebb-plugin-sso-custom@1.0.0 (installed, enabled)
    * nodebb-plugin-web-push@0.7.3 (installed, disabled)
    * nodebb-plugin-write-api-custom@5.8.12 (installed, enabled)
    * nodebb-rewards-essentials@1.0.1 (installed, enabled)
    * nodebb-theme-harmony@2.0.40 (installed, disabled)
    * nodebb-theme-lavender@7.1.18 (installed, disabled)
    * nodebb-theme-peace@2.2.39 (installed, disabled)
    * nodebb-theme-persona@14.0.16 (installed, disabled)
    * nodebb-theme-custom@1.0.0 (installed, enabled)
    * nodebb-widget-essentials@7.0.36 (installed, enabled)

    Nginx is being used to serve static assets and the config is the same as it was before the upgrade.

    Setup:
    Mongodb Atlas M40
    Redis is Heroku's own key value store premium 2
    Heroku -> 4 dynos (Standard-2X variant)
    In one dyno we have nginx + nodebb

    Any help would be greatly appreciated!
    Thanks in advance.

    L

    I can't disable it. The auto restart is a feature from our hosting provider

    0
  • L lassesuom@community.nodebb.org

    Hey, so after the upgrade the memory consumption has been creeping up at a fairly steady rate and exceeding 1GB of memory. Is this something to be expected from the version upgrade? Or could it be due to our own changes / plugins we have? Or due to scrapers perhaps?

    Upgrade happened on Monday 14th and before that the memory consumption was fairly steady at around 500MB. After the upgrade the memory consumption has been steadily going up during the day as seen in the pictures:

    Screenshot when the sudden increase was noticed:
    9119758a-4530-497e-a570-8f11d5d17015-image.png

    Here is last 7 days: (the falling edges are due to daily restart)
    a180780e-92ff-437f-bda7-b50c5265704b-image.png

    Screenshot from the /admin/development/info on the 23rd when it started also using swap.
    bdf69f0e-459b-4239-96f6-d8b89b7e7834-image.png

    Active plugins:
    * nodebb-plugin-s3-uploads@3.0.5 (installed, enabled)
    * nodebb-plugin-2factor@7.5.9 (installed, disabled)
    * nodebb-plugin-composer-default@10.2.47 (installed, enabled)
    * nodebb-plugin-dbsearch@6.2.13 (installed, enabled)
    * nodebb-plugin-emoji@6.0.2 (installed, disabled)
    * nodebb-plugin-emoji-android@4.1.1 (installed, disabled)
    * nodebb-plugin-gravatar@4.0.0 (installed, enabled)
    * nodebb-plugin-markdown@13.1.1 (installed, enabled)
    * nodebb-plugin-mentions@4.7.1 (installed, enabled)
    * nodebb-plugin-poll@3.2.5 (installed, disabled)
    * nodebb-plugin-question-and-answer@1.2.7 (installed, enabled)
    * nodebb-plugin-spam-be-gone@2.3.1 (installed, enabled)
    * nodebb-plugin-sso-custom@1.0.0 (installed, enabled)
    * nodebb-plugin-web-push@0.7.3 (installed, disabled)
    * nodebb-plugin-write-api-custom@5.8.12 (installed, enabled)
    * nodebb-rewards-essentials@1.0.1 (installed, enabled)
    * nodebb-theme-harmony@2.0.40 (installed, disabled)
    * nodebb-theme-lavender@7.1.18 (installed, disabled)
    * nodebb-theme-peace@2.2.39 (installed, disabled)
    * nodebb-theme-persona@14.0.16 (installed, disabled)
    * nodebb-theme-custom@1.0.0 (installed, enabled)
    * nodebb-widget-essentials@7.0.36 (installed, enabled)

    Nginx is being used to serve static assets and the config is the same as it was before the upgrade.

    Setup:
    Mongodb Atlas M40
    Redis is Heroku's own key value store premium 2
    Heroku -> 4 dynos (Standard-2X variant)
    In one dyno we have nginx + nodebb

    Any help would be greatly appreciated!
    Thanks in advance.

    julian@community.nodebb.orgJ

    Are you using the ActivityPub functionality? It might be related to that if only because that's what's new (among other smaller items) in v4.

    If you toggle the global activitypub switch off, does that make memory usage stabilize?

    Keep in mind that since you upgraded, ActivityPub is automatically disabled, so perhaps this is unrelated.

    1
    0
  • support@community.nodebb.orgS support@community.nodebb.org shared this topic
    System shared this topic
  • L

    ActivityPub hasn't been touched at all so it has been disabled the whole time.

    1
    0
  • L

    On a second note, is it normal behavior that redis has gotten 440k new keys in 4 days ? In last 7 days there has been less than 100 new topics created.. What kind of data is stored there? I guess sessions but what else?

    1
    0
  • baris@community.nodebb.orgB

    Are you using redis only or mongodb + redis? If redis only all forum data and sessions are stored there.

    You can use redis insight to see keys click the tree view to group them by namespace.

    1f073255-8d49-481c-82ac-3ef712d9fc78-image.png

    Link Preview Image
    1
    0
  • julian@community.nodebb.orgJ

    @lassesuom my guess is they're sess:* keys. They can balloon up if you have lots of guests *cough* AI scrapers.

    There used to be logic so that guests didn't receive a session key (since they didn't really need one for regular browsing) but it could be that that detection no longer works.

    1
    0
Anmelden zum Antworten

  • evanprodromou@socialwebfoundation.orgE

    Steps Forward in Long-form Text

    ActivityPub Test Kategorie implementers interop lft project standards
    1
    0 Stimmen
    1 Beiträge
    0 Aufrufe
    Niemand hat geantwortet
  • julian@community.nodebb.orgJ

    NodeBB v4.3.0 — Remote Categories are a go!

    ActivityPub Test Kategorie nodebb activitypub release 4.3.0
    17
    0 Stimmen
    17 Beiträge
    38 Aufrufe
    julian@community.nodebb.orgJ
    @esoteric_programmer@social.stealthy.club that's likely because GtS handles summary as a content warning for everything. The whole summary and content warning business is in flux right now, so hopefully a standard will be set soon.
  • C

    Cloudflare Caching

    ActivityPub Test Kategorie
    4
    0 Stimmen
    4 Beiträge
    2 Aufrufe
    C
    @julian No, Cloudflare can use the cache-control response header to determine how long to cache content, but not whether to cache content. Even if it was possible to cache based on response headers, it wouldn't be useful in this case; we want to ensure both that we're not serving user-specific content to guests (or other users) and that we're not serving guest-specific content to users, and the cache-control headers can only help with the former, not the latter.
  • hugh@ausglam.spaceH

    As far as I understand, most (all?) fediverse #ActivityPub software does not use the Client-to-server protocol from the specs (https://www.w3.org/TR/activitypub/#client-to-server-interactions) but rather use custom APIs instead.

    ActivityPub Test Kategorie activitypub
    49
    0 Stimmen
    49 Beiträge
    37 Aufrufe
    julian@community.nodebb.orgJ
    @smallcircles@social.coop personally, I feel that asking people to post or continue a discussion on another platform is a roadblock that shouldn't need to exist. You can't do it on Discourse, but in NodeBB you can "categorize" a topic, even if it came from the microblog-fedi. Similarly to how you can import a reply tree into another Mastodon instance. That's the difference, that the software should support something like this, although I get that that's not always important to every piece of software. Things get a little more confusing if a topic is already categorized, like a Lemmy/Piefed post in a community, so I expect some of that to change in the coming months. End of the day it would look something like "cross-posting" as currently exists on the threadiverse.
  • migelammon@community.nodebb.orgM

    Need solution that ensures PDFs can be previewed

    ActivityPub Test Kategorie
    3
    0 Stimmen
    3 Beiträge
    1 Aufrufe
    julian@community.nodebb.orgJ
    @eeeee is right, it's pretty much an arms race between those who want to get the file, and those who don't want the file downloaded. Keep in mind that if a browser is viewing the file, then it has de facto already downloaded the file. The "save as" functionality is mostly a convenience feature for you to re-locate the downloaded file elsewhere.
  • K

    I am wanting to use the forum on my pool site. But the forum adddress is going to be https://ktiptokyng.com/forum/

    ActivityPub Test Kategorie
    37
    0 Stimmen
    37 Beiträge
    6 Aufrufe
    K
    I am starting think this code is a piece of junk.
  • laz@community.nodebb.orgL

    How to mass edit post content?

    ActivityPub Test Kategorie
    4
    0 Stimmen
    4 Beiträge
    1 Aufrufe
    julian@community.nodebb.orgJ
    @Laz it was a typo, you can run the script with node script_name.js, which invokes the node.js interpreter.
  • julian@community.nodebb.orgJ

    Live testing of remote categories

    ActivityPub Test Kategorie activitypub nodebbactivityp
    63
    2
    0 Stimmen
    63 Beiträge
    959 Aufrufe
    julian@community.nodebb.orgJ
    @pfefferle@mastodon.social just wanted to poke you about this issue again. The latest updates to NodeBB now do a webfinger backcheck to ensure that the actor has a valid webfinger entry for their purported handle. If it does not, then the user is not properly created. Mastodon also does this. This check is probably for security as well as for preventing handle collisions. The multilingual plugin in conjunction with the ActivityPub plugin creates users that share the same handle, and that causes issues with federated content. For example, this article by @jonvt@vivaldi.com will load up just fine in Mastodon, but this japanese article by @akira@vivaldi.com will not, because that second article's attributedTo is https://vivaldi.com/ja/?author=176, which fails that check (the author's ID is actually https://vivaldi.com?author=176 as per the handle backcheck) cc @AltCode