dir.friendica.com cronjob not working #3
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
When I compare the stats for dir.friendica.com with those on my experimental server, I can tell that the cronjob is not doing the maintenance jobs.
Possibly the cronjob is not configured yet, if not can you report an issue with any error logs?
Seems to be running without error.
2014-09-26 23:46:53: Pulling from 0 remote targets.
2014-09-26 23:46:53: Pushing enabled, but no push targets.
2014-09-26 23:46:53: Syncing completed. Took 0 seconds.
2014-09-26 23:50:16: Pulling 0 items from queue.
2014-09-26 23:50:16: Pulling from 0 remote targets.
2014-09-26 23:50:16: Pushing enabled, but no push targets.
2014-09-26 23:50:16: Syncing completed. Took 0 seconds.
2014-09-27 00:00:02: Creating 10 maintainer threads for 80 profiles.
2014-09-27 00:00:02: Updating: http://www.net-lair.de/social/profile/adam
2014-09-27 00:00:02: Updating: http://thaibrew.net/profile/niels
2014-09-27 00:00:02: Updating: http://www.h5.gs/profile/ravindra
2014-09-27 00:00:02: Updating: http://www.net-lair.de/social/profile/leon
2014-09-27 00:00:02: Updating: http://friendica.freealways.com/profile/quadm
2014-09-27 00:00:02: Updating: https://www.myfilk.de/profile/kirstin
2014-09-27 00:00:02: Updating: http://friendica.dukun.de:1234/profile/owie2
2014-09-27 00:00:02: Updating: http://friendica.dukun.de:1234/profile/testor
2014-09-27 00:00:02: Updating: http://friendica.dukun.de:1234/profile/testator
2014-09-27 00:00:02: Pulling 0 items from queue.
2014-09-27 00:00:02: Pulling from 0 remote targets.
2014-09-27 00:00:02: Updating: http://lgy.fr/profile/thomas
2014-09-27 00:00:02: Pushing enabled, but no push targets.
2014-09-27 00:00:02: Syncing completed. Took 0 seconds.
[some updates omitted for brevity...]
2014-09-27 00:00:14: Update returns: 1
2014-09-27 00:00:14: Update returns: 1
2014-09-27 00:00:14: Update returns: 1
2014-09-27 00:00:14: Updating: http://hey-u.co.uk/profile/faguorobe
2014-09-27 00:00:14: Updating: https://betamax65.de/profile/betamax65
2014-09-27 00:00:15: Updating: http://www.allemannen-softworkx.de/profile/thilo
2014-09-27 00:00:15: Update returns: 1
2014-09-27 00:00:16: Update returns: 1
2014-09-27 00:00:16: Updating: https://friendica.libertypod.com/profile/pavi
2014-09-27 00:00:16: Update returns: 1
2014-09-27 00:00:17: Updating: https://rufpost.vega.uberspace.de/friendica/profile/matthias_eberl
2014-09-27 00:00:17: Update returns: 1
2014-09-27 00:00:19: Update returns: 1
2014-09-27 00:00:20: Update returns: 1
2014-09-27 00:00:20: Update returns: 1
2014-09-27 00:00:24: Maintenance completed. Took 22 seconds.
That looks good. Though the
Creating 10 maintainer threads for 80 profiles.suggests there may be some backlog. Was the cronjob recently started?No it seems to have been running all along,
Very interesting. In spite of this there seems to be a huge difference in profile information.
For example, currently dir.friendica.com has 10106 profiles. While my clone only has 2555 profiles.
Which is also visible in per-site health reports. (suggesting it is not due to downtime of the host)
http://dir-fc.oscp.info/health/148 (10 profiles)
http://dir.friendica.com/health/25 (72 profiles)
However the sites that have been probed for health at all is very different.
http://dir-fc.oscp.info/health?s=selfhost
http://dir.friendica.com/health?s=selfhost
I will perform another full sync with your server, to see if any missing profiles will be added or not.
In the meantime could you check:
/adminpage, what is the value for the maintenance backlog?.htconfig.php'ssite-healthandmaintenancearrays?health_scorevalues below yourremove_profile_health_thresholdin thesite-healthtable?Current maintenance backlog: 9220 entries
80 items per maintenance call.
/Things related to site-health monitoring.
$a->config['site-health'] = array(
//Wait for at least ... before probing a site again.
//The longer this value, the more "stable" site-healths will be over time.
//Note: If a bad (negative) health site submits something, a probe will be performed regardless.
'min_probe_delay' => 3_24_3600, // 3 days
//Probes get a simple /friendica/json file from the server.
//Feel free to set this timeout to a very tight value.
'probe_timeout' => 5, // seconds
//Imports should be fast. Feel free to prioritize healthy sites.
'skip_import_threshold' => -20
);
//Things related to the maintenance cronjob.
$a->config['maintenance'] = array(
//This is to prevent I/O blocking. Will cost you some RAM overhead though.
//A good server should handle much more than this default, so you can tweak this.
'threads' => 10,
//Limit the amount of scrapes per execution of the maintainer.
//This will depend a lot on the frequency with which you call the maintainer.
//If you have 10 threads and 80 max_scrapes, that means each thread will handle 8 scrapes.
'max_scrapes' => 80,
//Wait for at least ... before scraping a profile again.
'min_scrape_delay' => 3_24_3600, // 3 days
//At which health value should we start removing profiles?
'remove_profile_health_threshold' => -60
);
There are 277 entries less than -60 health_score; all are exactly -100
Just a thought - this is a shared host and you may not be used to dealing with them. If this is trying to do a lot of fetches in a single process or if processes aren't spaced out by delivery_interval, the cron jobs are likely getting killed by dreamhost. There is no warning or notification when this happens - the jobs just get killed. I went through a lot of trouble with polling and deliveries to get around shared host issues such as this. I need 4 or 5 seconds typically between processes and perhaps 3 network fetches maximum per process. It may not be this at all, but I just want to flag it.
That may well be the problem. Because 9.2K backlog is nearly the entire directory. Indeed I'm not running shared hosting. This is a virtual container running on my home server. Literally 2 feet away from it.
Can you have a look if there are many
Maintenance completed.Log entries?If the process gets killed these will probably be few.
If you know the details of what the limits are for your hosting, you may try to tweak the
threadsandmax_scrapesvalues. It currently would attempt 10 parallel scrapes, 8 scrapes per thread.Also which
delivery_intervaldo you mean?I ran on a home server in the states, but it's too expensive to do that in Australia. We pay for bandwidth here and it isn't cheap.
The friendica code may have changed but when I wrote it, we ran each delivery in a separate process (proc_run()) , and delayed starting the next one by $delivery_interval seconds. This is a system config which defaults to 2 seconds. This code may not be in the directory which didn't have to do any deliveries. In redmatrix (where we've implemented mirrored directories incidentally) we can batch these together (e,g, scrapes per thread). I can't do much in parallel.
I'm also running three redmatrix servers (one is a directory) and a friendica server (and about ten other webservers of different flavours) on this account. That's why I really want to find a way to offload the entire directory at some point. It's using resources I need for redmatrix.
Anyway, I can get away with perhaps 3 scrapes per thread, 1 in parallel, and the subsequent scrapes spaced out by 2-5 seconds. The job initiating them is probably going to get killed so it probably needs to restart from cron every ten minutes and pick random items to scrape so it doesn't hang on the same remote servers repeatedly and has a chance of getting some others each time.
It might be easier to send you a db dump in order to get through the backlog. Then the daily updates would probably be manageable.
We may need a more generic solution to this then. Some sort of job scheduler.
The backlog is not so much an issue for me, but it breaks the purpose of the maintenance features.
On dir.friendica.com will be many dead/removed profiles, which made the dir hard to use for me.
With the idea of being able to mirror a directory, other people will run into this with similar shared hosting restrictions.
So I think a generic and configurable utility that can spread out tasks like these is in order.
@friendica Does Red currently have job schedualing for shared host problems like this? If so, can you point me to related source code? Depending on how it looks we could package this as a seperate library usable for Red, Friendica and the Directory, with a collection of per-host recommended settings.
By default we turn all queue items into separate processes and run them one at a time separated by delivery_interval (which is in your site preferences). It's the same thing we did with the poller in Friendica. We had onepoll and onedeliver which handle one thing at a time. And we space these out by delivery_interval from the calling process. For Dreamhost I need to separate each process by 4-5 seconds to avoid having them killed for using too much resources. It only gives me about 10-15 concurrent processes before it starts killing things and I've got the friendica directory, a friendica node (that's mostly dormant but it's still handling traffic), four red servers and about fifteen other websites. Stuff gets killed all the time.
Red adds the ability to batch 'n' jobs together into a single process, so we can do for instance 3-4 queue items per process, but I can't really use it. That allows sites with lots of resources to keep things pumping through as fast as they can handle.
I would prefer to provide a DB dump of the directory and let somebody still working closely with the project take over the Friendica primary directory role (completely) so I can turn it off. I have no interest in making it scale or run on shared hosts. It's a global directory. It needs some decent resources, but the project was never able to provide these for me so I made do with what I had. It's grown since then and it's outgrown Dreamhost.
I've started a seperate conversation about the hosting issues for dir.friendica.com specifically https://fc.oscp.info/display/140839323114383093854c23b75714cb
For the time being I would like to focus on the coding problem though. The current directory has multiple jobs. Scrapes, polls and syncing. So spacing out would only work if there is a generic task tracker. I'll have a look at the polling code see what I can take from this.
Closed as stale.