Watch changes in a directory

Updated 20230123

There are at least two cases when watching for file changes, the private computer and the server.

The difference is the frequency of the changes. In a server you might have to decide what files you want to watch to not overload the server unnecessarily with thousands of changes every second. On your own machine changes may be a single number/second per directory. This latter case is what the following script can handle.

#!/bin/bash

#

#########

#   Name: 

#       pfdirwatch.sh

#

#   Version

#       20220402

#           Changes to make the display in the terminal window more readable when starting

#           w/o "n" or "y". The terminal will roll on, never be cleared, the printout limited

#           only by the number of lines available in the terminal window

#       20220207

#           added a log in /tmp. It will be deleted each time the machine reboots.

#           Each line in the log has a time assigned. 

#           NOTE: using parentheses in comments sometimes causes error in bash scripts!! It shouldn't but it does.

#           Removed parentheses!

#       20220130

#           bugfix. (The script only checked for changes in it's own working dir)

#       20220107

#           bugfix avoiding unnecessary message

#       20220101

#           Original version

#

#   Description:

#       This script takes two snapshots of the file system and then compares

#       them with respect to file time (changes) and existence (deleted or created).

#       Since it's rather primitive, changes can happen and vanish between the points

#       in time when the snapshots are taken. I use it to check for repetitive tasks

#       that changes log files. If changes occur, the task is running.

#       

#       The time between snapshots are lengthened by the $TIME_TO_READ bcs you need

#       to get a glimpse of what happened. 

#       

#   Usage:

#       I normally use it like:

#       pfdirwatch.sh  /home/xhosa/tools/scripts 10 n

#       NOTE:It will only work if you are in the same dir as the script when starting it!!

#           this will start the script looking like

#+++++++++++++++++++++

#We are in dir /home/xhosa/tools/scripts

#Command: /home/xhosa/tools/scripts/pfdirwatch.sh /home/xhosa/tools/scripts 5 n

#Pid: 598858

#+++++++++++++++++++++ 

#

#15:46:45

#

#       Setting the "n" to a "y" will run it once with only the time displayed and the next

#       loop the "y" will have been set to "n" and it will continue like above

#

#       Leaving out the "y"/"n" as the third argument makes the program run without the 

#        header. The terminal display will roll on, the TIME_TO_READ will not limit the time

#        you can view changes (they will be readable only limited by the number of lines available

#        in the terminal).

#

#########

# --TODO--: another variant of this script, logging to a file and starting tail on it. This way I'd have both live data

# available plus a log!!

#

# change this to what suits you. If you choose /tmp these files will be cleared when restarting

# otherwise you'll have to do that yourself.

TMP="/tmp/pfdirwatch"

FILE="/tmp/pfdirwatch.sh.log"

TIME_TO_READ=6

## creating log

if [ ! -f "$FILE" ]; then

    date +%Y-%m-%d_%H:%M:%S >> $FILE

    echo -e "Command:" $0 $1 $2 $3 >> $FILE

    echo "Logfile: " $FILE

fi

   echo -n "."  >> $FILE

   ## if pfdirwatch is a file, remove it, created the dir pfdirwatch

if [[ ! -d $TMP ]]

then                        # it's a file

    if [[ -f $TMP ]]

    then

        rm $TMP > /dev/null  2&>1

    fi

    mkdir $TMP

    chmod 0750 $TMP

else

    ## pfdirwatch IS a dir, remove all files in it

    for N in `ls -i -A /tmp/pfdirwatch/`; do ((loop++));done

    if(( $loop>0 ))

    then

        rm /tmp/pfdirwatch/* > /dev/null 2&>1

    fi

    

fi

## report on terminal will be readable for this no of seconds.

TIME_TO_READ=4

# asking for the number of arguments to the script, the script itself NOT counted!

if [ $# -lt 2 ]

then echo -e \\n"Usage '$0 folder waitseconds "\\n

exit 1

fi

## checking that argument 1 is a folder

if [ ! -d $1 ]

then

echo "Folder name is false!!"

exit 1

fi

## the "n" as argument 3 will make this report printout on the screen.

if [[ $3 == 'n' ]]

then

clear

echo -e \\n+++++++++++++++++++++

echo "This script's working directory: "`pwd`

echo "Command:" $0 $1 $2 $3

pid=$BASHPID

echo "Pid: "$pid

echo -e +++++++++++++++++++++ \\n

fi

echo `date +%H:%M:%S`

arg3="n"

# find only in this dir (no deeper) only regular files, stat lists %y(modification time) %s(ize) %n(ame) 

##find  $1 -maxdepth 1 -type f -not \( -name "$pid.lista.txt"  -o -name "$pid.listab.txt" \) -exec stat -c '%y %s %n ' {}  \;

find  $1 -maxdepth 1 -type f -not \( -name "$pid.lista.txt"  -o -name "$pid.listab.txt" \) -exec stat -c '%y %s %n ' {}  \; > $TMP/$pid.lista.txt

# list looks like:

# < 2011-08-10 12:47:43.000000000 +0200 196 ./tjosan 

# > 2011-08-10 12:49:42.000000000 +0200 203 ./tjosan 

# > 2011-08-10 12:49:42.000000000 +0200 0 ./124942 

# $2 (below) is the measure of time between now and then #the snapshots#, where eventual differences among the files will be

# discovered. This value is a gas pedal, sort of. Comment it or set it to 0 and you'll have live seconds working 

# in the printout.

sleep $2

##find  $1 -maxdepth 1 -type f -not \( -name "$pid.lista.txt" -o -name "$pid.listab.txt" \) -exec stat -c '%y %s %n ' {}  \;

find  $1 -maxdepth 1 -type f -not \( -name "$pid.lista.txt" -o -name "$pid.listab.txt" \) -exec stat -c '%y %s %n ' {}  \; > $TMP/$pid.listab.txt

# "sed -n '/^[><]/p'>" 

# From sed manual: 'By default sed prints all processed input (except input that has been modified/deleted

# by commands such as d). Use -n to suppress output, and the p command to print specific lines.' 

# These are decided by "'/^"=from the beginning of the line, "[> or <]"=lines beginning with either, "p"= pls 

# print these lines.

# array for diff-data:

diff $TMP/$pid.lista.txt $TMP/$pid.listab.txt|sed -n '/^[><]/p'

diff $TMP/$pid.lista.txt $TMP/$pid.listab.txt|sed -n '/^[><]/p'> $TMP/$pid.diffdata

# checking the block size of the $TMP/$pid.diffdata file

l=`ls -s  $TMP/$pid.diffdata|awk '{print $1}'`

# the ls command sys "no blocks are reserved for this file"

#if (( l == 0 ))

#then

    #this script| temp dir| time between snapshots

# exec  $0 $1 $2 $3

#fi

cat $TMP/$pid.diffdata|awk '{print $6 }'|uniq -d > $TMP/$pid.dupdata

cat $TMP/$pid.diffdata|awk '{print $6 }'|uniq -u $TMP/$pid.dupdata $TMP/$pid.uniqdata|sort 

# create indexed array. "-A" means "associative". Here are the snapshots compared.

# 

declare -a fs

declare -a ft

for udata in `cat $TMP/$pid.dupdata|awk '{print '$6'}'`

do

loop=0

while read ddata 

do

#echo "udata: "$udata

#echo "ddata: "`echo $ddata|awk '{print $6}'`

if [ "$udata" == "`echo $ddata|awk '{print $6}'`" ]

then

fs[$loop]=`echo $ddata|awk '{print $5}'`

# this is the file time:

ft[$loop]=`echo $ddata|awk '{print $3}'`

((loop++))

fi

done < $TMP/$pid.diffdata

if ! [ ${fs[0]} == ${fs[1]} ]

then

echo "Size of $udata has changed: ${fs[0]}  => ${fs[1]}";

arg3="y"

echo -e "Size of $udata has changed: ${fs[0]}  => ${fs[1]}"; >> $FILE

fi

if ! [ ${ft[0]} == ${ft[1]} ]

then

echo "Filetime of $udata has changed: "  `echo ${ft[0]}|cut -c -8`  " => "  `echo ${ft[1]}|cut -c -8` 

arg3="y"

        echo -e "Filetime of $udata has changed: "  `echo ${ft[0]}|cut -c -8`  " => "  `echo ${ft[1]}|cut -c -8` >>  $FILE

fi

done

for uqdata in `cat $TMP/$pid.uniqdata|awk '{print '$6'}'`

do

loop=0

while read ddata 

do

#echo "uqdata: "$uqdata

#echo "ddata: "`echo $ddata|awk '{print $6}'`

        

if [ "$uqdata" == "`echo $ddata|awk '{print $6}'`" ]

then

            DT=`date +%H:%M:%S`

if [ `echo $ddata|awk '{print $1}'` == "<" ]

then

echo ${DT}" Deleted file: "`echo $ddata|awk '{print $6}'` 

arg3="y"

                echo ${DT}" Deleted file: "`echo $ddata|awk '{print $6}'` >> $FILE

else

echo ${DT}" Added  file: "`echo $ddata|awk '{print $6}'`

arg3="y"

echo ${DT}" Added file: "`echo $ddata|awk '{print $6}'` >> $FILE

fi

fi

done < $TMP/$pid.diffdata

done

echo

# if changes are detected you'll have $TIME_TO_READ seconds to read about them

# the ls command sys "no blocks are reserved for this file"

if (( l == 0 ))

then

    #this script| temp dir| time between snapshots|"n"

exec  $0 $1 $2 $3

fi

    sleep $TIME_TO_READ

clear

# and here we go again, looping

exec  $0 $1 $2 $3

# - Example code #

# declare g as array and filling it directly from a command. Since there are

# five space-delimited fields in each stat-line and (in the current dir) 150 files

# we'll get 5 * 150 = 750 entries

# echo "1"

# declare -a g=(`find  . -maxdepth 1 -type f -not \(-name "$BASHPID.lista.txt"\) -exec stat -c '%y %s %n ' {}  \; |sort`)

# - end#

## - Example code #

## - Filling array from a file

# declare -a f

# cnt=0

# while read line

# do

#   f[$cnt]=$line

#   echo  ${f[$cnt]}

#   ((cnt++))

# done < lista.txt

## note: this means "${}" = execute, "#" = number of, f[] = the 'f' array, "@" all members of

## so this means: "what is the number of all items in the f-array?"

# echo "Number of f-items: "${#f[@]}

## end #

This is how it looks when running

and in fact, I've used it to keep an eye on the "screen_setup.sh_last_run.log" from the blogpost

https://a-zproj.blogspot.com/2021/12/xubuntux11-cant-remember-dual-monitor.html

It seems like that problem has been worked around nicely. That's me, enough irritation always causes me to use quick and dirty things instead of fixing the problem at the root.

Xubuntu/X11 can't remember a dual monitor configuration

(Updates are at the end of the post!) 

As soon as I leave my monitor for a cup of coffee or other everyday activities taking some ten minutes, when I come back my main screen is mirrored onto the secondary one. In the long run it's really irritating. But; this happens only when I'm not actively working. Sometimes I lean back to think and if I takes too long the configuration goes to mirroring.

I found a way to lessen the problem. I used two scripts (which perhaps ought to be merged but for this blog post they are what they are).  I thought I could put the setup script in .profile but I ran into problems, the login prompt didn't come up. I didn't explore that in any depth, I was in no mood to solve anything other that the screen problem.
I'll add one more thing: I run Xubuntu 20.04.02

First I need to know what my monitors are called in the X11 context, using xrandr:

snkdb@bender2:~$xrandr --listmonitors

Monitors: 2
0: +*DisplayPort-0 2560/597x1440/336+0+0  DisplayPort-0
1: +DisplayPort-1 1920/519x1200/324+2560+0  DisplayPort-1

Script "screen_setup.sh":
#!/bin/bash
### setting the correct configuration ###

# I had problems using "-" in the display port names. "DP-0" wasn't accepted.
# there's some confusion here...

DP0=DisplayPort-0
DP1=DisplayPort-1
# main screen: 2560x1440

function setup {
    xrandr --output $DP0 --auto --pos 0x0 --output $DP1 --auto --pos 2560x0
}

setup
# mostly for checking it worked as meant to 
date '+%Y-%m-%d %H:%M:%S' > <path>/screen_setup.sh_last_run.log
# END

Script "screen_setup_run_locally.sh":

#!/bin/bash ### running the configuration script (screen_setup.sh) repeatedly ###

# directions, how to use [in this case setting the wait time to 10s. You may have to experiment here]

if [ "$1" == "" ]; 

 then 

     echo "Pls enter the number of wait time seconds as an argument!"; 

     echo " like: 'screen_setup_run_locally.sh  10'<ENTER>"; 

     exit; 

fi 

# wait 10s

sleep $1 

# start this script in another process with the same wait time

exec $0 $1
#END

Then I created a launcher on the desktop. The command : <path>/screen_setup_run_locally.sh 10. After I have logged in I click it once and then screen_setup_run_locally.sh will stay put and do it's job. You can check it by starting ksysguard, choosing the Process Table tab, entering "screen_setup.sh" in the search line to the right of the "End Process"-button. In my environment it's a line looking:

sh  <your user name> <PID> <IO Write [~4k each 10s]> <Memory [~200k]> <shared Mem. value>

The result is: when I'm gone I won't find the configuration deteriorated as before when I come back after a pause.
The launcher icon on the desktop is possible to drag and drop onto the panel where I prefer to have it.

Update 20211206:

Very strange X11 behavior; setting up the scripts, xrandr --listmonitors resulted in DisplayPort-0 and 

DisplayPort-1. After the nightly backup, my machine shuts down. When I started it, the scripts didn't work, warning that the display ports didn't exist. xrandr --listmonitors  now resulted in ports names like DP-1 and DP-2! The format is changed and they now start at 1 instead of 0. What the f?! Why?
As of now I don't know why. Since the construction of these scripts isn't really my main goal in life I just added a second line to the screen_setup.sh:

xrandr --output $DP0 --auto --pos 0x0 --output $DP1 --auto --pos 2560x0

xrandr --output "DP-1" --auto --pos 0x0 --output "DP-2" --auto --pos 2560x0

One of them will always end up in a warning about not existing ports so the launcher will get an addition: <path>/screen_setup_run_locally.sh 10 &> /dev/null

This will take care about the annoying warning messages.

I'm pleased to present really good example of a kludge!!

Photography, birds in flight part 2. Change of camera Lumix FZ82 => Sony RX10 IV

Updated 20220828

The Lumix FZ82 has many good features but also severe limitations. Like any camera :)
Among the good things are 1200mm optical zoom, 4k, low weight. Among the bad things a totally worthless viewfinder. It's almost useless.

Since it's a "superzoom" it has a 1/2.3 sensor chip. The main consequence is that you have to shoot only in sunlight if the object is moving fast. Birds in flight normally do just that.

If the light is weak your images will be grainy.

I thought I'd try with a 1" chip camera so I bought a Lumix FZ1000 with 400mm optical zoom, thinking the increase in quality could balance out the longer zoom of the Lumix FZ82. I was prepared to be somewhat disappointed but "somewhat" was too much. The higher quality of about everything else wasn't enough.

For a long time I'd been interested in the Sony RX10 but the price was ~6 times higher compared to any camera I've owned up to that point. It was much money down the drain if it didn't worked out. I was *really* undecided for a long, long time but in the end I just ordered it and logged out. A big step for me! Phew!

It arrived and the feeling of quality in every detail was staggering! Smooth and heavy. In fact staggeringly heavy. I mount a red dot sight on my cameras for bird photography and the FZ82 weighs about 775g and the Sony 1318g [1lb, 11oz and 2lb, 14oz respectively] with these attached..

After using the lighter camera for years, the greater weight is still bothering me. Every time I take it up I think "Whew! Heavy!"

So far so good. In the autumn 2021 I went to a place where migratory birds pass every year. This was the ultimate test for the camera. The light was not good, cloudy and some rain. My old camera was *not* going to create great results with these preconditions, I knew that for sure.

After a couple of days I started to feel comfortable with the Sony, results were OK. Since it caught more light due to better lens and bigger chip, some margin for cropping emerged. What was disconcerting was the bokeh. Since the light was weak, the camera chose big apertures (I normally prefer short exposure times to get sharp birds :)  and the bokeh was extreme compared the the FZ82. 

To make a long story short: Yes, it worked, changing camera from Lumix FZ82 to Sony RX10 IV. The final proof is, I haven't used the Lumix since the Sony arrived. I'd say the "big step" was worth the money!
There are irritating things in this camera and there are equally irritating things in the Lumix, one just has to accommodate these by working around them or get used to them. I think that goes for any camera independent of prize.

Below images in cloudy conditions. White Wagtail (Motacilla alba), two images. Rather muscular guy! They are crops of about 25% of the original image.

White-tailed eagle (Haliaeetus albicilla) 
one image, taken when my gut feeling was "this is too far out" but it turned out to be an acceptable picture in my book. It's not a crop, this is as it came out of the camera in spite of the low light (very cloudy). NOTE: You have to look at in in full size on a computer, looking at it on your mobile is no use.

And lastly, the camera with the red dot sight. I have no problem mounting it, takes about 30 min. I never use the flash, I use my mobile for the very few times I need it. So, I glue with Bondic. There will forever be a mark but no functions of the camera are damaged. Other people may want to invest more but the problem is, fastening the sight in just the flash shoe is not enough. I *must* be fixated to the camera so it does not lose it's settings every time you transport it or use it. Horrible, I know :o)). New camera and all. But, a tool is a tool and it must do it's job.

I bought a strip of aluminium from Bauhaus, bent it, used a metal saw and a fine file to make it fit in the flashlight mount. Then I glued it to the lid of the flash. 

The last step is to adjust the red dot sight to hit the middle of the image in the view finder. When these two correspond exactly, you are done!

Using it, you look through the red dot sight ( NOTE: it MUST have green light as an option!! ) and as soon as you have the object in sight, you press the trigger button. That's all. There are caveats but you'll find them out yourself.

The sight cost about $30 from Alibaba.

A tip: I set my camera to 1/1000s or faster, always. That's the "S" setting that makes the camera always use this high speed setting.