Add advancedcontentfilter addon

MrPetovan commented

2018-04-17 04:24:33 +02:00

(Migrated from github.com)

Closes https://github.com/friendica/friendica/issues/4709

Let me explain.

I started working on this addon a couple weeks ago, at first it was just a simple POC, would the Symfony Expression Language (https://github.com/symfony/expression-language) allow Andy H3 to have his complex rule-based content filter system? It turned out that it worked like a charm almost at first try.

Now that the fun part of the work was over, I had to do the gruntwork: forms, lists, CRUD (Create, Read, Update, Delete), errors, and a hefty help page.

Halfway through the add form, I got hit by a severe lack of motivation. I had done this time and again, I didn't have to prove that I could do it, there was no challenge, no tingling to it. And yet it had to be done. How could I avoid to write Yet Another PHP Form?

Lately, I've been considering writing a SPA (Single Page Application) for Friendica, using the API as data provider, but the size of the task made it kind of overwhelming. Why not use the addon as an excuse to test new things, see how it goes?

Since it was a very simple use-case, only a single entity to manage without entity links, I looked for the simplest modern PHP backend and Javascript frontend. I had heard about the Slim framework (https://github.com/slimphp/Slim) through the Chronicle service (https://github.com/paragonie/chronicle), and I settled for VueJS (https://github.com/vuejs/vue) as it wasn't React and it was touted as simpler than AngularJS for apparently the same use.

The Slim framework really has a small footprint when combined with Friendica session, DB handler, authentication, with just under 70 lines of code for routing and error handling.

I was a little less happy with VueJS. The base framework isn't that big, but I had to (?) import vue-resource to perform the API calls, and it definitely felt like a duplication with our existing JQuery library.

The result, however, is beyond anything I could have hoped. Both the HTML for the settings page and the needed custom JS fit into less than 100 lines each. Once I got the hang of VueJS very peculiar way of manipulating the HTML, it became easy to add custom features on top of an example app I found online.

The whole thing feels fast and modern, and yes, it's about 1000 more files than really needed through the magic of Bower/npm package managers, but it's only hard disk, right?

I'm now considering using Slim to break down the massive include/api.php file into digestable classes while keeping the routing we kind of have.

Anyway, enjoy the snappiest part of the Friendica interface!

Depends on https://github.com/friendica/friendica/pull/4851

Reviewers: The real deal starts at b02724f867

Closes https://github.com/friendica/friendica/issues/4709 Let me explain. I started working on this addon a couple weeks ago, at first it was just a simple POC, would the Symfony Expression Language (https://github.com/symfony/expression-language) allow Andy H3 to have his complex rule-based content filter system? It turned out that it worked like a charm almost at first try. Now that the fun part of the work was over, I had to do the gruntwork: forms, lists, CRUD (Create, Read, Update, Delete), errors, and a hefty help page. Halfway through the add form, I got hit by a severe lack of motivation. I had done this time and again, I didn't have to prove that I could do it, there was no challenge, no tingling to it. And yet it had to be done. How could I avoid to write Yet Another PHP Form? Lately, I've been considering writing a SPA (Single Page Application) for Friendica, using the API as data provider, but the size of the task made it kind of overwhelming. Why not use the addon as an excuse to test new things, see how it goes? Since it was a very simple use-case, only a single entity to manage without entity links, I looked for the simplest modern PHP backend and Javascript frontend. I had heard about the Slim framework (https://github.com/slimphp/Slim) through the Chronicle service (https://github.com/paragonie/chronicle), and I settled for VueJS (https://github.com/vuejs/vue) as it wasn't React and it was touted as simpler than AngularJS for apparently the same use. The Slim framework really has a small footprint when combined with Friendica session, DB handler, authentication, with just under 70 lines of code for routing and error handling. I was a little less happy with VueJS. The base framework isn't that big, but I had to (?) import vue-resource to perform the API calls, and it definitely felt like a duplication with our existing JQuery library. The result, however, is beyond anything I could have hoped. Both the HTML for the settings page and the needed custom JS fit into less than 100 lines each. Once I got the hang of VueJS very peculiar way of manipulating the HTML, it became easy to add custom features on top of an example app I found online. The whole thing feels fast and modern, and yes, it's about 1000 more files than really needed through the magic of Bower/npm package managers, but it's only hard disk, right? I'm now considering using Slim to break down the massive `include/api.php` file into digestable classes while keeping the routing we kind of have. Anyway, enjoy the snappiest part of the Friendica interface! Depends on https://github.com/friendica/friendica/pull/4851 Reviewers: The real deal starts at https://github.com/friendica/friendica-addons/pull/586/commits/b02724f867b1f210d464c8e44d4a3f0d452b9d7f

MrPetovan commented

2018-04-17 05:20:28 +02:00

(Migrated from github.com)

Future improvements:

Get rid of vue-resource for the existing $.ajax. It will be less elegant, but the Ajax library redundancy really is ridiculous right now.
Add translated help pages in their own language subfolders.
Add popular examples
Make the language metadata into a neat array that can be easily queried in an expression to cover the langfilter usage (https://github.com/friendica/friendica/issues/4869)

Future improvements: - [x] Get rid of `vue-resource` for the existing `$.ajax`. It will be less elegant, but the Ajax library redundancy really is ridiculous right now. - [ ] Add translated help pages in their own language subfolders. - [ ] Add popular examples - [ ] Make the language metadata into a neat array that can be easily queried in an expression to cover the langfilter usage (https://github.com/friendica/friendica/issues/4869)

annando commented

2018-04-17 15:16:50 +02:00

(Migrated from github.com)

I wasn't able to check this, but: What type of line endings are having these files?

MrPetovan commented

2018-04-17 15:17:48 +02:00

(Migrated from github.com)

Probably the wrong one.

AndyHee commented

2018-04-17 16:16:42 +02:00

(Migrated from github.com)

I can see the dilemma in terms of the documentation.

At the moment, it's not suitable for users at all. To have such detailed and complex explantation is useful for those who want to dig deeper. But we need something of more clarity that basically centres around common examples.

I think, it might be better to have a separate very basic help page in addition to more advanced explanations. TBH, the current help looks formidable, may I say.

I am happy to write this basic explanation that would allow users to copy and paste a rule.

I checked out both PR and run composer install at the core.
friendica-addons/pull/586
friendica/pull/4851
But for some reason I cannot even "add" anything with the plus sign. Am I not doing this correctly?

I can see the dilemma in terms of the documentation. At the moment, it's not suitable for users at all. To have such detailed and complex explantation is useful for those who want to dig deeper. But we need something of more clarity that basically centres around common examples. I think, it might be better to have a separate very basic help page in addition to more advanced explanations. TBH, the current help looks formidable, may I say. I am happy to write this basic explanation that would allow users to copy and paste a rule. I checked out both PR and run composer install at the core. friendica-addons/pull/586 friendica/pull/4851 But for some reason I cannot even "add" anything with the plus sign. Am I not doing this correctly?

MrPetovan commented

2018-04-17 16:19:51 +02:00

(Migrated from github.com)

Which theme are you using? Do you have any error in the console? A modal form should appear when you press the plus button.

AndyHee commented

2018-04-17 16:22:22 +02:00

(Migrated from github.com)

Frio https://cloud.hubup.pro/s/bzE2bSmJWNePjnz

AndyHee commented

2018-04-17 16:27:02 +02:00

(Migrated from github.com)

Loading failed for the <script> with source “https://friendica.hubup.pro/addon/advancedcontentfilter/vendor/asset/vue/dist/vue.min.js”.
advancedcontentfilter:435
ReferenceError: Vue is not defined
[Learn More]
advancedcontentfilter.js:1:1

``` Loading failed for the <script> with source “https://friendica.hubup.pro/addon/advancedcontentfilter/vendor/asset/vue/dist/vue.min.js”. advancedcontentfilter:435 ReferenceError: Vue is not defined [Learn More] advancedcontentfilter.js:1:1 ```

MrPetovan commented

2018-04-17 16:44:34 +02:00

(Migrated from github.com)

Got it. There's a .gitignore file that prevented the add of the advancedcontentfilter/vendor/asset/vue/dist/ folder to the branch, so the library was never packaged with the addon.

Until I find a workaround, you need to run

cd addon/advancedcontentfilter
../../bin/composer.phar install

To get the required library.

Got it. There's a `.gitignore` file that prevented the add of the `advancedcontentfilter/vendor/asset/vue/dist/` folder to the branch, so the library was never packaged with the addon. Until I find a workaround, you need to run ``` cd addon/advancedcontentfilter ../../bin/composer.phar install ``` To get the required library.

AndyHee commented

2018-04-17 16:59:44 +02:00

(Migrated from github.com)

So far the composer only returns: "Nothing to install or update"

I'll try again tomorrow.

So far the composer only returns: "Nothing to install or update" I'll try again tomorrow.

MrPetovan commented

2018-04-17 17:02:40 +02:00

(Migrated from github.com)

Please delete your addon/advancedcontentfilter/vendor folder and try again.

Please delete your `addon/advancedcontentfilter/vendor` folder and try again.

AndyHee commented

2018-04-17 18:04:35 +02:00

(Migrated from github.com)

Got it! The modal form comes up properly now!

More testing tomorrow.

Got it! The modal form comes up properly now! More testing tomorrow.

🎉 1

annando commented

2018-04-17 18:08:45 +02:00

(Migrated from github.com)

I guess I will wait until @AndyHee finished his testing and @MrPetovan has clarified this vendor thing.

MrPetovan commented

2018-04-17 18:11:00 +02:00

(Migrated from github.com)

My current plan is to move the vue dependency to Friendica core, this is the only way to make it available through a git/composer update. In the process I'll probably drop vue-resource to use JQuery instead.

My current plan is to move the `vue` dependency to Friendica core, this is the only way to make it available through a git/composer update. In the process I'll probably drop `vue-resource` to use JQuery instead.

MrPetovan commented

2018-04-18 02:47:31 +02:00

(Migrated from github.com)

This has been done, the file counter is down to a, huh, reasonable number.

MrPetovan commented

2018-04-18 02:51:46 +02:00

(Migrated from github.com)

Turns out I should have done what I suggested @AndyHee and deleted my vendor folder completely, this would have saved me a headache.

AndyHee commented

2018-04-18 07:07:36 +02:00

(Migrated from github.com)

At the moment, I can't actually create a filter rule.

I get an error that the table called 'friendica.advancedcontentfilter_rules' doesn't exist.

Do I need to update the database structure? Or add the table manually?

At the moment, I can't actually create a filter rule. I get an error that the table called 'friendica.advancedcontentfilter_rules' doesn't exist. Do I need to update the database structure? Or add the table manually?

rabuzarus commented

2018-04-18 12:14:32 +02:00

(Migrated from github.com)

Lately, I've been considering writing a SPA (Single Page Application) for Friendica, using the API as data provider

Really cool. I support the idea.

> Lately, I've been considering writing a SPA (Single Page Application) for Friendica, using the API as data provider Really cool. I support the idea.

MrPetovan commented

2018-04-18 13:48:50 +02:00

(Migrated from github.com)

@AndyHee Friendica Core adds a new hook in the database structure definition for hooks to add tables. When you enable the addon, a database structure update is performed to add the relevant table to your database. Can you please try to "Reload you Active Addons", or disabling/enabling the advancedcontentfilter addon again? The table should be added to your database during the addon install, unless your Friendica MySQL user can't create tables?

AndyHee commented

2018-04-18 13:54:31 +02:00

(Migrated from github.com)

That sounds easy. I'll try this.

AndyHee commented

2018-04-18 16:21:06 +02:00

(Migrated from github.com)

This works, @MrPetovan !

I created two filters, 1: author on twitter AND keyword in body 2: author on twitter AND keyword NOT in body. Everything was hidden accordingly.

Installation seemed more sane now. Last time there was a timeout error in the browser after activating the addon. Nothing this time.

I'll do some real-life tests now and tomorrow. So far a very impressive addon!

This works, @MrPetovan ! I created two filters, 1: author on twitter AND keyword in body 2: author on twitter AND keyword NOT in body. Everything was hidden accordingly. Installation seemed more sane now. Last time there was a timeout error in the browser after activating the addon. Nothing this time. I'll do some real-life tests now and tomorrow. So far a very impressive addon!

🎉 1

MrPetovan commented

2018-04-18 16:35:55 +02:00

(Migrated from github.com)

Thanks, I guess the timeout was caused by the forced database structure update on addon activation, you may have missed a past update, and the catch up took too long for a browser request.

AndyHee commented

2018-04-18 16:54:13 +02:00

(Migrated from github.com)

I see, thanks.

I noticed on the help page there is some inconsistency in the "Supported variables" section. It says "author-name" for instance but "author_name" appears to be the deployed format for the variables.

I see, thanks. I noticed on the help page there is some inconsistency in the "Supported variables" section. It says "author-name" for instance but "author_name" appears to be the deployed format for the variables.

MrPetovan commented

2018-04-18 17:08:07 +02:00

(Migrated from github.com)

Indeed, the correct name to use in the expressions is author_name because otherwise the hyphen is interpreted as a minus operator. The hyphen comes from the database field name.

Indeed, the correct name to use in the expressions is `author_name` because otherwise the hyphen is interpreted as a minus operator. The hyphen comes from the database field name.

MrPetovan commented

2018-04-19 02:34:44 +02:00

(Migrated from github.com)

I noticed on the help page there is some inconsistency in the "Supported variables" section. It says "author-name" for instance but "author_name" appears to be the deployed format for the variables.

Fixed.

> I noticed on the help page there is some inconsistency in the "Supported variables" section. It says "author-name" for instance but "author_name" appears to be the deployed format for the variables. Fixed.

👍 1

AndyHee commented

2018-04-23 12:07:53 +02:00

(Migrated from github.com)

I somehow still experience high resource consumption when switching branches.

System Load: 57%
Ram Usage: 92%
Disk Usage: 36%

php 24.7% 1.21 GB root (4)
mysqld 7.2% 99.44 MB mysql (1)
php 4.6% 28.16 MB www-data (3)
php-fpm7.0 3.7% 47.9 MB www-data (3)

Could this be related to the DB table being removed?

I somehow still experience high resource consumption when switching branches. ``` System Load: 57% Ram Usage: 92% Disk Usage: 36% php 24.7% 1.21 GB root (4) mysqld 7.2% 99.44 MB mysql (1) php 4.6% 28.16 MB www-data (3) php-fpm7.0 3.7% 47.9 MB www-data (3) ``` Could this be related to the DB table being removed?

MrPetovan commented

2018-04-23 13:57:09 +02:00

(Migrated from github.com)

I don't think so, I never experienced the removal of the table when I switched branches. I still have the advancedcontentfilter_rules table in my database at the moment even though I don't have the task/add-advancedcontentfilter-addon branch active.

Even then, it shouldn't take so long to drop a table with no foreign keys.

I don't think so, I never experienced the removal of the table when I switched branches. I still have the `advancedcontentfilter_rules` table in my database at the moment even though I don't have the `task/add-advancedcontentfilter-addon` branch active. Even then, it shouldn't take so long to drop a table with no foreign keys.

AndyHee commented

2018-04-23 15:31:23 +02:00

(Migrated from github.com)

Thanks for shedding light on this.

It must be something else going on with my database, very likely not related to the dropping of the table. I rebooted the server and the resource consumption is still very high.

Thanks for shedding light on this. It must be something else going on with my database, very likely not related to the dropping of the table. I rebooted the server and the resource consumption is still very high.

MrPetovan commented

2018-04-23 16:28:00 +02:00

(Migrated from github.com)

I'm sorry I can't troubleshoot further, how's the addon otherwise?

AndyHee commented

2018-04-23 16:45:23 +02:00

(Migrated from github.com)

Not to worry!

Yes, it worked very well from what I have observed so far. There was no unexpected behaviour so ever in the course of using it.

I think, this addon will be a brilliant feature!

Not to worry! Yes, it worked very well from what I have observed so far. There was no unexpected behaviour so ever in the course of using it. I think, this addon will be a brilliant feature!

MrPetovan commented

2018-04-23 16:47:33 +02:00

(Migrated from github.com)

I think so too, I'm planning more work to expose more data to the filter as well. The goal for this addon is for it to be able to completely cover the use cases of the other content filter addons (blockem, langfilter, nsfw) as a one-stop content filter shop.

❤️ 1

AndyHee commented

2018-04-23 17:07:13 +02:00

(Migrated from github.com)

This sound great!

Imagine this: In the long, users would be able to add filer rules on the fly. "Filter posts like this" would populated a range input of fields, such as the post's assumed language, author's name, key terms, etc. on which basic rules can be added... OK, just a vision.

This sound great! Imagine this: In the long, users would be able to add filer rules on the fly. "Filter posts like this" would populated a range input of fields, such as the post's assumed language, author's name, key terms, etc. on which basic rules can be added... OK, just a _vision._

MrPetovan commented

2018-04-23 17:09:11 +02:00

(Migrated from github.com)

This is especially tricky, even GMail doesn't reliably identify the pattern used when using "Create a filter from those emails".

AndyHee commented

2018-04-23 17:27:43 +02:00

(Migrated from github.com)

True. But the easy of such input fields you have in this GMail "Filter messages like this" function is rather persuasive for average users.

Combining this with the power of procmail style filter recipe would cater for both advanced and basic users.

True. But the easy of such input fields you have in this GMail "Filter messages like this" function is rather persuasive for average users. Combining this with the power of procmail style filter recipe would cater for both advanced and basic users.

MrPetovan commented

2018-04-23 17:30:29 +02:00

(Migrated from github.com)

Ah, you mean having a "simple mode" where you fill a few form fields and it generates the expression? It is possible.

AndyHee commented

2018-04-23 17:45:42 +02:00

(Migrated from github.com)

Yes, but I guess some fields could be populated quite easily like "author_link".

Also an advantage of such a simple (activation) mode could be that you don't need to go into the settings. Instead you could just bring it up via the post's menu where you star or delete posts.

Yes, but I guess some fields could be populated quite easily like "author_link". Also an advantage of such a simple (activation) mode could be that you don't need to go into the settings. Instead you could just bring it up via the post's menu where you star or delete posts.

MrPetovan commented

2018-04-23 17:46:51 +02:00

(Migrated from github.com)

Ah, the dream of making accessible a heavy abstraction...

😆 1

MrPetovan commented

2018-05-01 00:59:18 +02:00

(Migrated from github.com)

Let's move this forward.

👍 1

AlfredSK commented

2018-05-01 12:19:24 +02:00