shreadme
Table of Contents
SHELF Rationale and Practice
1 Scope
This document supports the SHELF Standard. In addition to providing rationale for the standard, it documents the function families which support an implementation of the standard.
IT is TAKING over the job from AUXLIB. All families are as before,
except unique auxlib
functions (no other family) are now in the
util group (a collection without a family). This note will
disappear when the turn-over is complete
The SHELF sets a practice for organizing shell functions into
re-usable libraries. As a library, the shelflib
contains the
standard-supporting function families. The library's source is
embedded in this document as tangled fragments. The ancillary
tanglelib
which is not part of the SHELF manages the production and
installation of the shelflib
2 Families
2.1 om – the object-model, or object-method for naming functions
2.2 report – or assert for verifying function arguments
A reporting function may be used in any shell function which needs to assert its arguments as to number, type of file, or string length, and any assertion you can support with a function.
This is a typical use of one of these functions as an assertion:
report_not{something} $1 && return N
where N is usually a 1, … the number of an error return from the user function, and "something" is readable, writable, executable, ….
Here is a sample function.
function report_notreadable { [[ -r $1 ]] && return 1; report_usage $1 is NOT readable }
Notice the usage above. It's a positive statement to assert an argument isn't something, and return a failure when that's the case. So, the user function lines up as many assertions on it's arguments as needed. The first one to fail produces the required message and returns from the function with a message on stderr from report_usage. Otherwise the function executes as desired.
2.3 shdoc – a subset, the Shell DOCumentation standard
These functions encourage a programmer to write a brief introductory user guide to each function.
There are functions here to collect the available function descriptions.
The format was developed in Java, Perl, and Python. The format here is familiar to persons who've used javadoc, etc…
The shd_… functions in this family, set a standard for shell doclib capture. Precedent for the practice was first established by Javadoc and more recently in Perldoc and Pydoc.
At the moment (2/13/16), the online references to shdoc are few and disjoint. The newest reference to a shdoc package is targeted at the TeX community.
2.4 trace – function execution tracing, and
The tracing feature of these shell functions uses nil in a function body after some assertion validation using the report functions. While it's not dependent on all assertions passing, it seems useful to dispose of the assertions, which announce a failure, and then announce the function body is ready to work.
Tracing my be enabled or disabled, by nil or nil. Either of them reset the function behavior to either nil, or simply return, respectively.
At bottom, the tracing logs it's calls to a per-user file:
HOME/lib/trace.log
and echo's each call to the stderr, which may be separately saved or altogether ignored:
... 2> trace.err # say, or ... 2> /dev/null # the "bit-bucket"
In any case, while tracing is enabled, the nil function inspects the latest entry in the log to see if it's time to post a new HOUR mark. I felt it useful to collect usage at this level, anticipating collecting function usage statistics. On reflection, the necessary test for the last record might be more work than necessary. I've done some work elsewhere, to collect the hourly work by number of command executions.
For each call, the trace log collects two function names: the current function and it's calling function. This enables a user to construct a calling tree from the log.
2.5 util – a few programming functions, .e.g. foreach
3 Functions
By family, where the utilies aren't a family, but the missing pieces. Nothing in the library uses a function not in the library.
3.1 om family
3.1.1 om_iam
The om_iam function is the main user function. It must be called by the family's _init function. When called, it creates these functions, where subfunctions are appended to the family name. e.g. family_list:
- family – uses om_generic to supply help and firsttime functionality
- _firsttime – executed the first time the family is loaded, whose default action is to supply the copyright notice for the family.
- _list – if not already defined, produces the list of family member functions
- _vars – if not already defined, produces the names of family member shell variables
- _help – if not already defined, produces a help message, defaluting to the family list and family variables
3.1.2 om_generic
This function offers it's users three standard features:
- any function in such a family may be invoked:
family subfun arg ... # rather than family_subfun arg ...
the simple reason being for command line use; the apace bar is much more convenient than the underscore. having supplied this gloss, it's necessary to point out to use the underscore name when using such a function in other functions, simply for name-discovery.
- the default behavior for the head of the family, the family-name
itself, is identical to the the use:
family help
- invokes the function:
family firsttime
only the first time the family or one of it's members is called thru the family name. This is a way to display a copyright notice only once, the default behavior. In this sence, the first time means "in the process of the current shell".
copyright_om () { comment "Copyright (C) 2014-2016, JYATL - Just Yet Another Testing Lab"; comment "mailto: mcgowan (at) alum DOT mit DOT edu"; } om_tally () { trace_call $*; om_list | sort | uniq -c } om_help () { echo to see OM functions, om_list; } om_list () { sfg _ | awk -F_ '{ printf "object\t%s\nmethod\t%s\n", $1, $2 }' } om_init () { declare -f om_iam >/dev/null || { comment $(myname) OM_IAM is NOT a function; return 1; } om_iam } om_iam () { : user identifies themself as an OM object; set $(myname 2); set ${1%_init}; report_notcalledby ${1}_init && return 1; : create the OM function eval "$1 () { om_generic $1 \$*; }" : every family has a list of members local l=${1}_list; declare -f $l > /dev/null 2>&1 || { eval "$l () { (sfg ^${1}_; sfg _${1}$) 2>/dev/null; }"; } : every family may have shell varialbles local v=${1}_vars; local V=$(UC $1) declare -f $v > /dev/null 2>&1 || { eval "$v () { set | grep ^${V}_ 2>/dev/null; }"; } : and a default help function when one doesnt exist local h=${1}_help; declare -f $h > /dev/null 2>&1 || { eval "$h () { echo $1 functions, variables:; ($v; $l) | sed 's/^/ /'; }"; } : shows Copyright when library is sourced. local f=${1}_firsttime; eval "$f () { copyright_${1}; unset $f; }"; : and displays the help ${1}_help } om_generic () { : ~ fun { sub arg ... }; : prefereably executes FUN_SUB arg ...; : or FUN_HELP; : ====================================; local fun=$1; shift; : only when a function is first use thru the model; declare -f ${fun}_firsttime > /dev/null && ${fun}_firsttime; [[ $# -lt 1 ]] && { ${fun}_help 2> /dev/null; return }; local f=${fun}_$1; shift; declare -f $f >/dev/null 2>&1 || { ${fun}_help 2> /dev/null; return }; $f $* }
3.2 report family
3.2.1 reportlib, main
And here is the report
function family.
First a copyright, then the initialization function. Since this is a family, the only necessary call is to om_iam
copyright_report () { comment "Copyright (C) 2015-2016, JYATL - Just Yet Another Testing Lab"; comment "mailto: mcgowan (at) alum DOT mit DOT edu"; } report_init () { om_iam } @include report.1 report_init 1>&2
3.2.2 reportlib, working
Then here are the working functions.
function report_emptyfile { [[ -s $1 ]] && return 1; report_usage File: is empty } function report_isfile { [[ -f $1 ]] || return 1; report_usage $1 IS a file and should not be } function report_needcount { [[ $2 -ge $1 ]] && return 1; usage need at least $1 arg/s $(shift 2; echo $*) } function report_notbase { [[ $(basename $PWD) == "$1" ]] && return 1; comment change to "$1" directory } function report_notblockspecial { [[ -b $1 ]] && return 1; report_usage $1 is NOT a blockspecial file } function report_notcalledby { set ${1:-/dev/null} $(myname 3) non-Existant-function; [[ $2 == $1 ]] && return 1; report_usage was NOT called by $1. } function report_notcharacterspecial { [[ -c $1 ]] && return 1; report_usage $1 is NOT a characterspecial file } function report_notdirectory { [[ -d $1 ]] && return 1; report_usage $1 is NOT a Directory } function report_notexecutable { [[ -x $1 ]] && return 1; report_usage $1 is NOT an executable file } function report_notexisting { [[ -e $1 ]] && return 1; report_usage $1 is NOT an existing file } function report_notfunction { set ${1:-/dev/null}; isfunction $1 && return 1; report_usage $1 is NOT a Function } function report_notgroupiseuid { [[ -G $1 ]] && return 1; report_usage $1 is NOT a file with groupiseuid } function report_notlargeenough { [[ $2 -ge $1 ]] && return 1; report_usage $1 $(echo 1 $1, 2 $2 $(shift 2; echo $*)) } function report_notnonzerostring { [[ -n $1 ]] && return 1; report_usage $1 is NOT a nonzerolengthstring } function report_notowneriseuid { [[ -O $1 ]] && return 1; report_usage $1 is NOT a file with owneriseuid } function report_notpipe { [[ -p /dev/stdin ]] && return 1; report_usage is NOT reading a pipe } function report_notreadable { [[ -r $1 ]] && return 1; report_usage $1 is NOT a readable file } function report_notsetgroupid { [[ -g $1 ]] && return 1; report_usage $1 is NOT a file with setgroupid } function report_notsetuserid { [[ -u $1 ]] && return 1; report_usage $1 is NOT a file with setuserid } function report_notsocket { [[ -S $1 ]] && return 1; report_usage $1 is NOT a socket } function report_notstickybitset { [[ -k $1 ]] && return 1; report_usage $1 is NOT a file with its stickybitset } function report_notsymboliclink { [[ -L $1 ]] && return 1; report_usage $1 is NOT a symboliclink } function report_notwritable { [[ -w $1 ]] && return 1; report_usage $1 is NOT a writable file } function report_notzerolengthstring { [[ -z $1 ]] && return 1; report_usage $1 is NOT a zerolengthstring } function report_usage { comment USAGE $(myname 3): $* } report_functions () { set | awk '$2 ~ /\(\)/ { print $1 }' | grep '^report_' } report_nofilefrom () { set $(eval $*) /dev/null $*; [[ -f $1 ]] && return 1; shift; report_usage \'$*\' did not return a file name. } report_notargcount () { [[ $2 -ge $1 ]] && return 1; report_usage need at least $1 arg/s: $(shift 2; echo $*) } report_notcommand () { ignore type -a $1 && return 1; report_usage $1 is NOT a command } report_notfile () { [[ -f $1 ]] && return 1; report_usage $1 is NOT a file } report_notfilegreaterthanzero () { [[ -s $1 ]] && return 1; report_usage $1 is NOT a filegreaterthanzero } report_nottrue () { eval "$@" && return 1; report_usage $@ FAILED }
3.3 shdoc family
3.3.1 shdoc, shd_with
The functions here operate on shell functions, and operate similar to the three mentioned above: Java, Perl, and Python. The two most useful functions, shdoc and shd_with produce a function whose body is the collected comments in what I'm calling shdoc, or shell doclib comment format:
@include shdoc.1 @include shdoc.2 @include shdoc.3 shdoc_init () { om_iam }
shdoc () { : this is a shell doclib "shdoc" comment; : an shdoc comment is the first ":"-origin lines; : in the shell function, the rest being the executable.; for f in ${*:-$(myname)}; do shd_each $f; done } shd_with () { : removes shd_less functions from shdoc results; : creating function {name}_doc for function "name"; shdoc $* | awk ' # { print "DEBUG: ", fcount, last > "/dev/stderr"; } $1 ~ /function/ { fcount++; last = $0; next; } $1 ~ /}/ && last { next } last { print last; last = ""; fcount = 0; } fcount == 0 { print } ' } function shdoc_doc { : this is a shell doclib "shdoc" comment; : an shddoc comment is the first ":"-origin lines; : in the shell function, the rest being the executable.; }
Note the result. The shdoc_doc function is the first colon-delimited lines of the shdoc function.
shd_with is the most useful of the bunch, as it says by removing shd_less functions from shdoc results.
3.3.2 shd_each
To complete the picture, here is shd_each. It's notable in that it insists on being called by shdoc.
shd_each () { report_notfunction $1 && return 1; report_notcalledby shdoc && return 2; trace_call $*; echo "function ${1}_doc {"; declare -f $1 | awk ' NR > 2 { if ( $1 !~ /^:/ ) exit else print } '; echo "}" }
Note shd_each produces the result for only one function, if only to print a single function body; it insists on being called by shdoc.
Here's a question: why doesn't shd_each have any shdoc comments?
3.4 trace family
function trace_base { printf "TRACE %s ( %s )\n" ${FUNCNAME[1]} "$*" 1>&2 } function trace_basic { set /tmp/$USER.y; declare -f trace_base | sed 's/_base/_call/' > $1; . $1 } function trace_debug { comment $PWD $(myname 2) $*; trap read debug } function trace_fbdy { printf "TRACE %s ( %s )\n" ${FUNCNAME[1]} "$*" 1>&2; nlevel=${#FUNCNAME[@]}; for ((i = 1; i < $nlevel; i++)) do set ${BASH_LINENO[$i-1]} ${FUNCNAME[$i]} ${BASH_ARGV[0]}; [[ $1 -lt 0 ]] && set 0 $2 /dev/null; trace_level $* $i $nlevel; done } function trace_help { echo trace functions:; sfg trace_ | sed 's/^/ /' } function trace_init { declare -f om_iam >/dev/null || { comment $(myname) OM_IAM is NOT a function; return 1; } om_iam } function trace_isOFF { local tr=$(trace_state); trace_off; eval $*; eval $tr } function trace_easy { echo $* 1>&2 } function trace_isOKto { [[ $1 == trace_${1/trace_} ]] || echo $1 } function trace_level { line=$(sed -n ${1}p $3); printf " %d / %d \t%-14s\t%4d: %s\n" $4 $5 "$2()" $1 "$line" 1>&2 } function trace_on { eval "trace_call () { trace_stderr \"\$@\"; }"; # for the moment, disable this tracing mechanism # eval "trace_call () { return; }"; trace_set trace_on trace_off } function trace_off { eval "trace_call () { return; }"; trace_set trace_off trace_on } function trace_set { case $1.$2 in trace_on.trace_off | trace_off.trace_on) setenv TRACE_STATE ${1}; setenv TRACE_NOT ${2} ;; trace_on.*) trace_set $1 trace_off ;; trace_off.*) trace_set $1 trace_on ;; esac } function trace_show { : show a basic, easy trace_call; declare -f trace_easy | sed 's/easy/call/' } function trace_state { echo $TRACE_STATE } function trace_summary { set ~/lib/trace.log; trace_call $*; cat $1 | awk ' $1 ~ /TRACE/ && ( $3 ~ /fm/ || $3 ~ /@/) { count[$2 "\t" $4]++; next } $1 ~ /HOURLY/ { for (c in count) { printf "SUM\t%6d\t%s\n", count[c], c } print $0 nc = 0 delete count } ' } function trace_tmpoff { stack_push $(trace_state); trace_off; $*; trace_set $(stack_pop) } function trace_tmpon { stack_push $(trace_state); trace_on; $*; trace_set $(stack_pop) } function trace_toggle { trace_set $TRACE_NOT $TRACE_STATE; $(trace_state); trace_state }
3.5 utility functions
Note the shell idiom in the shd_top function. It's a convenient shorthand to list a collection of similar names. In this case, this shdlib conventionally defines functions whose name begins with shd_.
shd_top () { echo shdoc shd_{oc,each,with,test} } shd_test () { : 1. test default, NO arguments, then; : 2. stub out test all functions in the library; : 3. demonstrate ignore subsequent comments; : 4. test shd_with; : 5. test shd_each defends against non-shdoc call; shdoc; :; set $(sfg shd_); : doit shdoc $*; shd_with $*; shd_each $*; declare -f shd_test | grep -v '^ *:' 1>&2 }
The standard practice for such a family has the executable om_iam
which establishes the function family of functions sharing the common
preface.
For example:
$ shd top # calls shd_top, returning shdoc shd_oc shd_each shd_with shd_test
4 Tangling
4.1 The installation script
install_bin shelf
4.2 the top shelf
@include util.0 @include report.0 @include shdoc.0 @include trace.0 @include om.0 @include shelf.0
shelf_init () { om_init om_iam report_init shdoc_init trace_init } shelf_init 1>&2
5 Verification
This section supplies and documents the functions and procedures to
verify the shelflib
conformance to this standard. A bit
chicken-or-egg type of problem. That's the nature of the beast.
6 References
If reading on paper, this document is found on-line at http://mcgowans.org/marty3/commonplace/software/shdoc.html
- the SHELF Standard
- my Commonplace Book
Email: mcgowan@alum.mit.edu
Twitter: @applemcg