Another sample, on a tricky texture with unshaded black areas (256 colour version didn't have darker reds available).
New script has been parameterized and you can set a floor on the source so the filter will soften & shade dark patches like this while retaining dark edges.
before/after:
i.png
o.png
Most of the intermediate files get emitted to tmp/ but a few are retained at the working directory to provide feedback on the floor and edge detection steps and guide settings. The filter steps themselves don't need adjusted.
Invoke the script like this:
./texfilter.sh WALL49_1.png
Arguments:
FLOOR sets a lower limit the source pixel intensity for the edge detector, so transitions to dark areas won't be considered as large transitions. Set with care and watch 'floor.png' to make sure the image doesn't saturate out.
floor.png
GAIN is used to amplify pixel variations in the source image before edge detection. Useful on low-contrast images otherwise leave it alone.
GAM? sets the gamma (power) curve for the edge detector. higher values will retain only the sharpest details. Lower values will retain more detail in other parts but will limit opportunities for shading with new colours.
MAG? sets the final overall magnitude (brightness) of the edge detection. If GAM? seems correct but the edge detect looks too dark or too bright you can fudge it into the right range as a final step with this.
The 3 and 5 refer to the 3x1 and 5x1 soften filters used to add shading. Each one has its own edge detector and filter, so you may have to adjust each differently to get the edge detector maps to look sensible and get decent results out.
You can monitor the edge detection output by looking at edge3.png and edge5.png. Try to get the brightest pixels fairly bright, and the darkest areas black, with some kind of variation in between. If you're getting muddy grey all over, the GAM or FLOOR is wrong. If you're getting mostly black with white dots, reduce GAM and play with MAG for that edge detector. If you are having trouble picking out detail from a low-contrast image, increase GAIN. Likewise, perhaps reduce it if the source texture is already high contrast.
edge3.png
edge5.png
You'll have to try different settings with different textures to get optimal results but these png files are a fair guide to the results you'll get out.
FLOOR="0.3"
GAIN="2.0"
GAM3="0.75"
MAG3="1.05"
GAM5="1.0"
MAG5="1.5"
mkdir tmp
# todo: parameterize everything properly
convert $1 -fx 'p' i.png
# split r,g,b
convert i.png -fx 'r' tmp/r.png
convert i.png -fx 'g' tmp/g.png
convert i.png -fx 'b' tmp/b.png
convert tmp/r.png -fx 'flr='$FLOOR'; max(p,flr)' tmp/r0.png
convert tmp/g.png -fx 'flr='$FLOOR'; max(p,flr)' tmp/g0.png
convert tmp/b.png -fx 'flr='$FLOOR'; max(p,flr)' tmp/b0.png
convert tmp/r0.png tmp/b0.png tmp/g0.png -fx 'max(max(u[0],u[1]),u[2])' floor.png
# 1x3 edge detect x:y
convert tmp/r0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[-1,0]*-1.0) + (p[0,0]*2.0) + (p[1,0]*-1.0)) * gf) / 2.0), pf) * mf' tmp/rxe3.png
convert tmp/g0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[-1,0]*-1.0) + (p[0,0]*2.0) + (p[1,0]*-1.0)) * gf) / 2.0), pf) * mf' tmp/gxe3.png
convert tmp/b0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[-1,0]*-1.0) + (p[0,0]*2.0) + (p[1,0]*-1.0)) * gf) / 2.0), pf) * mf' tmp/bxe3.png
convert tmp/r0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[0,-1]*-1.0) + (p[0,0]*2.0) + (p[0,1]*-1.0)) * gf) / 2.0), pf) * mf' tmp/rye3.png
convert tmp/g0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[0,-1]*-1.0) + (p[0,0]*2.0) + (p[0,1]*-1.0)) * gf) / 2.0), pf) * mf' tmp/gye3.png
convert tmp/b0.png -virtual-pixel tile -fx 'mf='$MAG3'; gf='$GAIN'; pf='$GAM3'; pow(abs((((p[0,-1]*-1.0) + (p[0,0]*2.0) + (p[0,1]*-1.0)) * gf) / 2.0), pf) * mf' tmp/bye3.png
convert tmp/rxe3.png tmp/gxe3.png tmp/bxe3.png -fx 'max(max(u[0],u[1]),u[2])' tmp/xedge3.png
convert tmp/rye3.png tmp/gye3.png tmp/bye3.png -fx 'max(max(u[0],u[1]),u[2])' tmp/yedge3.png
convert tmp/xedge3.png tmp/yedge3.png -fx 'max(u[0],u[1])' edge3.png
# 1x3 filter x:y
convert tmp/r.png -virtual-pixel tile -fx '(((p[-1,0]*1.0) + (p[0,0]*2.0) + (p[1,0]*1.0)) / 4.0)' tmp/rxf3.png
convert tmp/g.png -virtual-pixel tile -fx '(((p[-1,0]*1.0) + (p[0,0]*2.0) + (p[1,0]*1.0)) / 4.0)' tmp/gxf3.png
convert tmp/b.png -virtual-pixel tile -fx '(((p[-1,0]*1.0) + (p[0,0]*2.0) + (p[1,0]*1.0)) / 4.0)' tmp/bxf3.png
convert tmp/r.png -virtual-pixel tile -fx '(((p[0,-1]*1.0) + (p[0,0]*2.0) + (p[0,1]*1.0)) / 4.0)' tmp/ryf3.png
convert tmp/g.png -virtual-pixel tile -fx '(((p[0,-1]*1.0) + (p[0,0]*2.0) + (p[0,1]*1.0)) / 4.0)' tmp/gyf3.png
convert tmp/b.png -virtual-pixel tile -fx '(((p[0,-1]*1.0) + (p[0,0]*2.0) + (p[0,1]*1.0)) / 4.0)' tmp/byf3.png
# 1x5 edge detect x:y
convert tmp/r0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[-2,0]*-1.0) + (p[-1,0]*-1.0) + (p[0,0]*4.0) + (p[1,0]*-1.0) + (p[2,0]*-1.0)) * gf) / 4.0), pf) * mf' tmp/rxe5.png
convert tmp/g0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[-2,0]*-1.0) + (p[-1,0]*-1.0) + (p[0,0]*4.0) + (p[1,0]*-1.0) + (p[2,0]*-1.0)) * gf) / 4.0), pf) * mf' tmp/gxe5.png
convert tmp/b0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[-2,0]*-1.0) + (p[-1,0]*-1.0) + (p[0,0]*4.0) + (p[1,0]*-1.0) + (p[2,0]*-1.0)) * gf) / 4.0), pf) * mf' tmp/bxe5.png
convert tmp/r0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[0,-2]*-1.0) + (p[0,-1]*-1.0) + (p[0,0]*4.0) + (p[0,1]*-1.0) + (p[0,2]*-1.0)) * gf) / 4.0), pf) * mf' tmp/rye5.png
convert tmp/g0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[0,-2]*-1.0) + (p[0,-1]*-1.0) + (p[0,0]*4.0) + (p[0,1]*-1.0) + (p[0,2]*-1.0)) * gf) / 4.0), pf) * mf' tmp/gye5.png
convert tmp/b0.png -virtual-pixel tile -fx 'mf='$MAG5'; gf='$GAIN'; pf='$GAM5'; pow(abs((((p[0,-2]*-1.0) + (p[0,-1]*-1.0) + (p[0,0]*4.0) + (p[0,1]*-1.0) + (p[0,2]*-1.0)) * gf) / 4.0), pf) * mf' tmp/bye5.png
convert tmp/rxe5.png tmp/gxe5.png tmp/bxe5.png -fx 'max(max(u[0],u[1]),u[2])' tmp/xedge5.png
convert tmp/rye5.png tmp/gye5.png tmp/bye5.png -fx 'max(max(u[0],u[1]),u[2])' tmp/yedge5.png
convert tmp/xedge5.png tmp/yedge5.png -fx 'max(u[0],u[1])' edge5.png
# 1x5 filter x:y
convert tmp/r.png -virtual-pixel tile -fx '(((p[-2,0]*1.0) + (p[-1,0]*2.0) + (p[0,0]*3.0) + (p[1,0]*2.0) + (p[2,0]*1.0)) / 9.0)' tmp/rxf5.png
convert tmp/g.png -virtual-pixel tile -fx '(((p[-2,0]*1.0) + (p[-1,0]*2.0) + (p[0,0]*3.0) + (p[1,0]*2.0) + (p[2,0]*1.0)) / 9.0)' tmp/gxf5.png
convert tmp/b.png -virtual-pixel tile -fx '(((p[-2,0]*1.0) + (p[-1,0]*2.0) + (p[0,0]*3.0) + (p[1,0]*2.0) + (p[2,0]*1.0)) / 9.0)' tmp/bxf5.png
convert tmp/r.png -virtual-pixel tile -fx '(((p[0,-2]*1.0) + (p[0,-1]*2.0) + (p[0,0]*3.0) + (p[0,1]*2.0) + (p[0,2]*1.0)) / 9.0)' tmp/ryf5.png
convert tmp/g.png -virtual-pixel tile -fx '(((p[0,-2]*1.0) + (p[0,-1]*2.0) + (p[0,0]*3.0) + (p[0,1]*2.0) + (p[0,2]*1.0)) / 9.0)' tmp/gyf5.png
convert tmp/b.png -virtual-pixel tile -fx '(((p[0,-2]*1.0) + (p[0,-1]*2.0) + (p[0,0]*3.0) + (p[0,1]*2.0) + (p[0,2]*1.0)) / 9.0)' tmp/byf5.png
# selective mixer 3x1:5x1
convert tmp/rxe5.png tmp/rxf3.png tmp/rxf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/rxc3.png
convert tmp/gxe5.png tmp/gxf3.png tmp/gxf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/gxc3.png
convert tmp/bxe5.png tmp/bxf3.png tmp/bxf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/bxc3.png
convert tmp/rye5.png tmp/rxf3.png tmp/ryf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/ryc3.png
convert tmp/gye5.png tmp/gxf3.png tmp/gyf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/gyc3.png
convert tmp/bye5.png tmp/bxf3.png tmp/byf5.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/byc3.png
# selective mixer 1x1:3x1
convert tmp/rxe3.png tmp/r.png tmp/rxc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/rxc.png
convert tmp/gxe3.png tmp/g.png tmp/gxc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/gxc.png
convert tmp/bxe3.png tmp/b.png tmp/bxc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/bxc.png
convert tmp/rye3.png tmp/r.png tmp/ryc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/ryc.png
convert tmp/gye3.png tmp/g.png tmp/gyc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/gyc.png
convert tmp/bye3.png tmp/b.png tmp/byc3.png -fx '((0.0+u[0]) * u[1]) + ((1.0-u[0]) * u[2])' tmp/byc.png
#convert tmp/rxe3.png tmp/rxc.png tmp/rye3.png tmp/ryc.png -fx '(u[0] < u[2]) ? u[1] : u[3]' tmp/rc.png
#convert tmp/gxe3.png tmp/gxc.png tmp/gye3.png tmp/gyc.png -fx '(u[0] < u[2]) ? u[1] : u[3]' tmp/gc.png
#convert tmp/bxe3.png tmp/bxc.png tmp/bye3.png tmp/byc.png -fx '(u[0] < u[2]) ? u[1] : u[3]' tmp/bc.png
# inverse selective mixer x:y
convert tmp/rxe3.png tmp/rxc.png tmp/rye3.png tmp/ryc.png -fx '(((0.001+u[2]) * u[1]) + ((0.001+u[0]) * u[3])) / ((0.001+u[0])+(0.001+u[2]))' tmp/rc.png
convert tmp/gxe3.png tmp/gxc.png tmp/gye3.png tmp/gyc.png -fx '(((0.001+u[2]) * u[1]) + ((0.001+u[0]) * u[3])) / ((0.001+u[0])+(0.001+u[2]))' tmp/gc.png
convert tmp/bxe3.png tmp/bxc.png tmp/bye3.png tmp/byc.png -fx '(((0.001+u[2]) * u[1]) + ((0.001+u[0]) * u[3])) / ((0.001+u[0])+(0.001+u[2]))' tmp/bc.png
convert tmp/rc.png tmp/gc.png tmp/bc.png -channel RGB -combine o.png
[EDIT]
The previous floor tile converts reasonably well with these settings:
FLOOR="0.0"
GAIN="2.0"
GAM3="0.5"
MAG3="1.25"
GAM5="0.8"
MAG5="1.5"
You do not have the required permissions to view the files attached to this post.