Went and Broke Federation. Kinda.
-
Sometime yesterday my forum stopped receivng (or accepting?) new remote topics or posts. I'd spent time yesterday playing with plugins and permissions, so I've been rolling back those changes this morning, but it hasn't resulted in the resumption of content flow.
The only thing the logs indicate is that the forum was receiving dislikes and didn't know how to handle them (at some point, I denied the federated user group the ability to downvote), but last night it just stopped complaining about this. I assumed at this point maybe I'd been spamming some kind of error at the other servers and got blocked or something, but I can still chat (two way) with my remote accounts, and remote accounts can see my posts on the forum. I don't receive any replies, though, weirdly enough, I do receive likes/upvotes.
Are there other logs I should be looking at to make sense of this?
-
S support@community.nodebb.org shared this topic
-
@Kichae test against a new account on ActivityPub.academy, when you tag your account it should notify you and show up in world.
Does that happen?
If not, run NodeBB in dev mode (so AP logging is output to console) and try again. Let me know what NodeBB says when it receives the activity.
-
Finally got around to actioning this. Pinging myself from activitypub.academy got a 403 response: "Sending failed (htps://wanderingadventure.party/inbox responded with status 403 Forbidden)", and generated the following in the console log while in dev mode:
2025-02-24T15:58:18.922Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T15:58:18.923Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T15:58:18.927Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB" 2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB" 2025-02-24T15:58:19.117Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB" 2025-02-24T15:58:19.370Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification 2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed. 2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload... 2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed. 2025-02-24T15:58:19.377Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed. 2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Asserting 1 actor(s) 2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Processing https://activitypub.academy/users/braulus_aelamun 2025-02-24T15:58:19.383Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun 2025-02-24T15:58:19.390Z [4566/131736] - warn: Route requested but not found: /api/v1/timelines/public?limit=40 2025-02-24T15:58:19.415Z [4566/131736] - warn: Route requested but not found: /api/v1/streaming/public 2025-02-24T15:58:19.799Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/followers 2025-02-24T15:58:19.802Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/following 2025-02-24T15:58:20.310Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed. 2025-02-24T15:58:20.317Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517 2025-02-24T15:58:20.777Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517 contains no context. 2025-02-24T15:58:20.819Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found. [api] Exception caught, error with stack trace follows: Error: [[error:no-privileges]] at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async Promise.all (index 0) at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3) at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19) at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)No such errors occur, however, if I follow the user account:
2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:21:10.504Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:21:10.958Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification 2025-02-24T16:21:10.959Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed. 2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload... 2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed. 2025-02-24T16:21:10.971Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed. 2025-02-24T16:21:10.972Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)... 2025-02-24T16:21:10.978Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved. 2025-02-24T16:21:11.014Z [4566/131736] - verbose: [activitypub/send] https://activitypub.academy/inbox 2025-02-24T16:21:11.466Z [4566/131736] - verbose: [activitypub/send] Successfully sent Accept to https://activitypub.academy/inboxNor if I send a direct message:
2025-02-24T16:25:27.790Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification 2025-02-24T16:25:27.792Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed. 2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload... 2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed. 2025-02-24T16:25:27.798Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed. 2025-02-24T16:25:27.802Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.Following the activitypub.academy account also seems to work fine:
2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:27:54.808Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:27:54.811Z [4566/131736] - verbose: [activitypub/inbox.delete] Object (https://mastodon.social/users/goma0) does not exist locally. Doing nothing. 2025-02-24T16:27:55.216Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification 2025-02-24T16:27:55.217Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed. 2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload... 2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed. 2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check failed, stripping object down to id. 2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed. 2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed. 2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)... 2025-02-24T16:27:55.237Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved.But when I post publicly from it, the messages bounce from my nodebb site, once again with a 403 error:
2025-02-24T16:30:11.720Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key 2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification 2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed. 2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload... 2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed. 2025-02-24T16:30:11.728Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed. 2025-02-24T16:30:11.732Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed. 2025-02-24T16:30:11.741Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026 2025-02-24T16:30:12.190Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026 contains no context. 2025-02-24T16:30:12.203Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found. [api] Exception caught, error with stack trace follows: Error: [[error:no-privileges]] at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async Promise.all (index 0) at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3) at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19) at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)While skimming the output stream, I'm also noticing a lot of entries from Mastodon servers reporting 410 errors:
2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification... 2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://mastodon.social/users/pathinisca#main-key 2025-02-24T16:24:36.874Z [4566/131736] - verbose: [activitypub/get] https://mastodon.social/users/pathinisca#main-key 2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Received 410 when querying https://mastodon.social/users/pathinisca#main-key 2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Error received: Gone 2025-02-24T16:24:36.889Z [4566/131736] - verbose: [activitypub/verify] Failed, key retrieval or verification failure.It's not clear to me what these are, though, as when I try to boost content or post from my main Mastodon account, which is being followed by my nodebb account, I get the 403 error reported a the top of this post.
Is there some kind of conflict with respect to file ownership that's causing this? Or is this about the fediverse pseudo-user's permissions to access different categories? Because the linux user running nodebb should have full ownership rights to the folder, but it's possible I've broken that somehow. Meanwhile, I've absolutely been playing with category access rights, but I've reset them all since this started.
