Not logged in. · Lost password · Register
Forum: General Help and Support Plugins Plugin Wishlist RSS
Dead link checker and link status
Plugin request :)
Page:  1  2  next 
Avatar
krang #1
User title: Awesome
Member for a month · 4 posts · Location: Dimension X
Group memberships: Members
Show profile · Link to this post
Subject: Dead link checker and link status
Aloha,

I am looking for someone who could create a plugin for me. I offer 40 USD (Please provide a price for this kind of customization if you think it is too low) (payment in Bitcoins).

I would like to have a status indicator after each external link on my wiki.
Each link need to be checked after the page is fully loaded. So checking status of external link cannot delay page loading.
Once wiki page is fully loaded each external link is checked and got own status.

First status is "checking" (once page is loaded, this status shows, next to external link symbol, till page is checked, so this is first temporary status, default one, before we get main info about offline or online) - this status icon should be yellow dot -
Second is "online" - page status ok, server response 200 - icon green dot -
Third is "offline" - broken link, server response 404 - red dot -

I think php curl or php wget function can be used for this.
I just know that in standard curl it can be done by running command:
curl -s -o /dev/null -I -L -w "%{http_code}\n" http://google.com
-s - Silent or quiet mode.
-o - Write output to <file> instead of stdout.
-I - Fetch the headers only.
-L - If the server reports that the requested page has moved to a different location (indicated with a Location: header and a 3XX response code), this option will make curl redo the request on the new place.
-w - Make curl display information on stdout after a completed transfer.

I was looking for some info about php-curl and maybe this function could be helpful:

curl_getinfo — Get information regarding a specific transfer
curl_getinfo ( resource $ch [, int $opt ] ) : mixed
where parameter
opt may be one of the following constants:
CURLINFO_RESPONSE_CODE - The last response code.

Thanks in advance!
Avatar
andi (Administrator) #2
User title: splitbrain
Member since May 2006 · 3391 posts · Location: Berlin Germany
Group memberships: Administrators, Members
Show profile · Link to this post
check https://www.wikimatrix.org/consultants/dokuwiki/programmin… to find someone to commission a custom plugin.

However it will be more expensive than you think. I guess implementing what you want would take about 4 hours for someone familiar with DokuWiki and Javascript. With typical market rates this would be something between 250 to 500 Euro.
Read this if you don't get any useful answers.
Lies dies wenn du keine hilfreichen Antworten bekommst.
Avatar
krang #3
User title: Awesome
Member for a month · 4 posts · Location: Dimension X
Group memberships: Members
Show profile · Link to this post
Ok. Thank you for the link. That is expensive. I will need to find other solution. Anyway. Thanks.
Avatar
cziehr #4
Member since Jan 2011 · 512 posts · Location: 10119 Berlin
Group memberships: Members
Show profile · Link to this post
I wrote a bash script for me which is based on https://www.techpaste.com/2012/01/bash-script-check-applic…

I added some code before the code from the link, which crawls the wiki-pages and isolates external links from them. This list of urls is given to the mentioned script.

Important: My script doesn't write the output to the wiki-page where the link was found, but writes it to a text-file or sends an email. But it would be possible to create the output from the script in a way that it is wiki-code an can be pasted on a wiki-page for further inspection.

Because you wrote

curl -s -o /dev/null -I -L -w "%{http_code}\n" http://google.com

I assume you're using some kind of a linux/unix-based OS, which is required for executing the script.
If you're interested, I would change my script that it can be used by everyone, and add some comments to understand whats going on ;-)
This post was edited on 2019-01-20, 23:00 by cziehr.
Avatar
krang #5
User title: Awesome
Member for a month · 4 posts · Location: Dimension X
Group memberships: Members
Show profile · Link to this post
This looks nice. I need some time to analyze this. Also one programmer offer his time to work on this so I am in the middle of negotiation.
Hmm, if I understand good it's hard to avoid JavaScript, building this kind of plugin. But your solution is closest to solution without JavaScript I think.

I am using VPS with Linux so I have access to execute scripts.

Thank you for your post.

EDIT:

ohh one user already create plugin like this: https://www.dokuwiki.org/plugin:xtern
I will test it and if everything will be ok author of plugin will get that 40 USD for great solution.
Thank you. I will test it over the weekend.
This post was edited on 2019-01-22, 20:19 by krang.
Avatar
andyboeh #6
Member since Sep 2014 · 4 posts
Group memberships: Members
Show profile · Link to this post
The plugin "linksenhanced" (https://www.dokuwiki.org/plugin:linksenhanced) now offers this option, too.
Avatar
cziehr #7
Member since Jan 2011 · 512 posts · Location: 10119 Berlin
Group memberships: Members
Show profile · Link to this post
A solution in a plugin is much better than my scripted link-overview-wiki-page. I assume posting my script isn‘t necessary any more. I will also use one of this plugins from now on.
Avatar
turnermm (Moderator) #8
Member since Oct 2009 · 4490 posts · Location: Canada
Group memberships: Global Moderators, Members, Super Mods
Show profile · Link to this post
Krang offered a fee and contacted me offline, but I thought that this was a good general use plugin and so spent some time over last weekend making a plugin which automatically detects broken links.  No markup is needed.  No fee required.  In its current form it replaces the gray DokuWiki external link icons with a green one for valid links and a red one for broken links, in keeping with the  scheme for internal links.  The image as displayed here enlarges the sizes both of images and texts, not too pretty, but you get the idea.
Myron Turner
github: https://github.com/turnermm
plugins, templates: http://www.mturner.org/devel
The author has attached one file to this post:
xtern.png 10.9 kBytes
You have no permission to open this file.
This post was edited on 2019-01-23, 15:02 by turnermm.
Avatar
cziehr #9
Member since Jan 2011 · 512 posts · Location: 10119 Berlin
Group memberships: Members
Show profile · Link to this post
Thank you for your work!

Would it be possible to create a page in the admin-menu where all broken links are listed? This would be really great for maintenance!
Avatar
andyboeh #10
Member since Sep 2014 · 4 posts
Group memberships: Members
Show profile · Link to this post
Of course, but that sounds like quite some work since you need to aggregate all links somehow - I do have some ideas on how to implement that, though.

EDIT: I just implemented it in linksenhanced. Beware, every link checked causes a new request to your DokuWiki installation for getting the link status, so it might take a while to load!
This post was edited on 2019-01-25, 23:52 by andyboeh.
Avatar
krang #11
User title: Awesome
Member for a month · 4 posts · Location: Dimension X
Group memberships: Members
Show profile · Link to this post
Ok I tested it on my wiki in Tor network and it works very well. If someone would like to check the results it is here: http://6s3qqbqolj3qkj76.onion (you need Tor Browser to reach that site)

I modified code a little to allow curl recognize .onion domains. Modification was made in action.php file.
Curl section should look like this:

$ch = curl_init($url);
        //curl_setopt($ch, CURLOPT_CAINFO, __DIR__ . "/certs/cacert.pem");
                curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
                curl_setopt($ch,CURLOPT_FOLLOWLOCATION);
                curl_setopt($ch, CURLOPT_MAXREDIRS, 5);
                curl_setopt($ch,CURLOPT_TIMEOUT,10);
                curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME);
                curl_setopt($ch, CURLOPT_PROXY, "localhost:9050");
                $output = curl_exec($ch);
                $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
                curl_close($ch);
                echo "$httpcode";
        return 1;

so changes were made in line, un-comment and set to false:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
and added
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME);
curl_setopt($ch, CURLOPT_PROXY, "localhost:9050");

Thank you all for your help. DokuWiki community is now one of the best for me.
Avatar
turnermm (Moderator) #12
Member since Oct 2009 · 4490 posts · Location: Canada
Group memberships: Global Moderators, Members, Super Mods
Show profile · Link to this post
There is also an issue on Windows (and possibly elsewhere) where curl doesn't have access to the ssl certificiates.  A solution is provided here:
  https://stackoverflow.com/questions/14078182/openssl-file-…
Simply download the certificates from this site:
   https://curl.haxx.se/docs/caextract.html
And reference the pem file in action.php:
       curl_setopt($ch, CURLOPT_CAINFO, __DIR__ . "/certs/cacert.pem");
      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
I will eventually build this fix into the plugin.
Myron Turner
github: https://github.com/turnermm
plugins, templates: http://www.mturner.org/devel
Avatar
turnermm (Moderator) #13
Member since Oct 2009 · 4490 posts · Location: Canada
Group memberships: Global Moderators, Members, Super Mods
Show profile · Link to this post
In reply to post #9
Temporarily, there is a script in the scripts directory named chk_links.php.  Put this in the lib/exe directory and run it as follows:
  php chk_links.php

It will output a list of all broken links: the link text, the status code, and the wiki page where it occurs.  If eventually I create an admin class for the plugin, it will go there. You can get the script by updating the plugin.
Myron Turner
github: https://github.com/turnermm
plugins, templates: http://www.mturner.org/devel
This post was edited on 2019-01-28, 02:22 by turnermm.
Avatar
cziehr #14
Member since Jan 2011 · 512 posts · Location: 10119 Berlin
Group memberships: Members
Show profile · Link to this post
Quote by turnermm:
Temporarily, there is a script in the scripts directory named chk_links.php.  Put this in the lib/exe directory and run it as follows:
  php chk_links.php

It will output a list of all broken links: the link text, the status code, and the wiki page where it occurs.  If eventually I create an admin class for the plugin, it will go there. You can get the script by updating the plugin.

Thank you, this sounds great, I will try. The script would be fully sufficient for me.
Avatar
turnermm (Moderator) #15
Member since Oct 2009 · 4490 posts · Location: Canada
Group memberships: Global Moderators, Members, Super Mods
Show profile · Link to this post
I've been working on an update which contains an admin panel where broken links can be checked and edited.  Significant improvements over earlier.

https://github.com/turnermm/xtern/archive/save_broken.zip
Myron Turner
github: https://github.com/turnermm
plugins, templates: http://www.mturner.org/devel
This post was edited on 2019-02-13, 16:47 by turnermm.
Close Smaller – Larger + Reply to this post:
Verification code: VeriCode Please enter the word from the image into the text field below. (Type the letters only, lower case is okay.)
Smileys: :-) ;-) :-D :-p :blush: :cool: :rolleyes: :huh: :-/ <_< :-( :'( :#: :scared: 8-( :nuts: :-O
Special characters:
Page:  1  2  next 
Go to forum
Imprint
This board is powered by the Unclassified NewsBoard software, 20150713-dev, © 2003-2015 by Yves Goergen
Current time: 2019-02-16, 16:50:14 (UTC +01:00)