Marty's 2014 Software Dairy

solidifying the implementation of my markdown utilities is my prime focus these days: I'm "off the clock" on the org mode todo's since after writing requirements, which outline a manual page or two, forces me to pick up the working pieces in a more concise manner. i.e one that can be rendered in manual pages, commands in bin directories and file formats a user can work with. So, at this moment, my challenge is standardizing the working format for the installation. I've been lax in using [github][], though most of the functionality is stored in ~/Dropbox/git, and therefore available on any platform I choose to enable. But the real sharing will take place on [github][].

Now it's to begin thinking about the current disparate sources:

• commonplace – where we are now
• Family – my diary and life history.
• html – a staging area for public exposure, and
• git – the host for [github][] sharing
• /usr/local – locally installed tooling,

/usr/local is from the root directory; the others are based at $(home), that being $HOME/Dropbox.

I invented this dotty function to make the relative directory hunting quite a bit easier:

dotty () { set – ${1:-bin}; trace_call $*; d=.; while true; do ignore pushd $d; rooted && { popd; return }; ignore popd; [ -d $d/$1 ] && { echo $d/$1; return }; d=$d/..; done }

this function returns the dotted path to the nearest named directory, defaulting to bin, traversing up the directory list

for putting a file in any of the directories in the specified list:

• /usr/local – installation script according to bitmeld principles
• html – all ~ references
  • external – presumed to exist
  • file:// – need to be recovered, copied or linked, to $(home)-relative names, where non-commonplace, non-Family, are rooted relative to, say file
  • $(dotty) names – installed according to [bitmeld][].
• installing any HTML file in these places requires two actions:
  • adjusting any references in the html file,
  • installing otherwise un-installed objects according to this rule.

A first step today, is a new function: mark2html. It notes a hierarchy of suffixes for Markdown source, now mk, mkt, mkd, txt, and src. Mark2html expects each file-suffix has an accompanying functions suf_**2html**, which knows how to convert it's content into marked-down html. The _mkt suffix is a special case, or really not quite, since it's the intermediate file from a Table of Contents (see mark_toc) conversion.

But, for example: if there is a file.**src** and a file.**html** in the same directory, then a user-supplied function: _src2html knows how to manage the conversion.

Time Machine doesn't back up dot files (e.g. .profile). Since I've been keeping my function history in dotted file names, e.g. $(home)/etc/lib/.20140606, and I inadvertently deleted a bunch of one-line files in attempting to remove a few empty files, the search for a back-up in Time Machine proved fruitless. Links (i.e. ln'ed files) to the rescue.

It was an easy exercise to create a new linked directory – ofun in smartfiles. And link the entire history in etclib to ofun. And then move (mv) the linked files to the un-dotted name. All that was left was to modify the T function, the smartfile name for today_s function. All this was precipitated by my perceived need for a Y function. Read _smartfiles to see how that's done.

Today, a tentative, but likely committed DECISION, as to point #5 above, on the tug of war between emacs and the command line: since the purpose of shell functions is to enable the command-line-user, I'll do the serious testing on the command line. Test development and unit testing can happen in a shell window in emacs. But, for command history, and re-running what are hoped stable tests, I'll use the command line.

I am noticing a different use of history in the emacs shell window. Rather than scroll thru the history with the line navigation commands, since these conflict with emacs buffer commands – maybe there is a way around even this possible confusion – I prefer the relative history number: e.g. !-N, often N=2, since a pair of commands are often sufficient to enable a function and test it. Or use the interline edit mode: !!:s/this/that

For the first practice of Saturday's requirements I'll tackle the [commonplace][] book itself.

putting markdown to use now, first thoughts: the first question is "why bother with the -t option. why not simply generate a gratuitous TOC?

With some work, I've moved dotty functionality to the m4 macros. It's in [toc][]. This bring up the issue, it almost looks as if we need a dotty _here_file, either to resolve dotty's without an m4 filter, and on installation, to adjust the resolution. Think about it.

OK, I did. Let's plan to replace {DOTTY html} with the results of a call to $(dotty html) in this source. In two modes. in this directory, it's "in-place", so, any ./../../html string in this directory will be replaced the the equivalent dotty call when moving the file. Let's try that now:

On Friday and Saturday I finally got busy with shellfunctions I'm nearing a re-publication. The challenge will be to offer a free upgrade to those who've purchased it already, while upping the price by a buck.

The little editing of the last week was to make the "diff"s fit on the page of a book, not requiring the layout, or on-its-side view for text wider than about 80 columns.

Also, in a little clean-up today, here's a few new functions to backup files (signalled by a *~ name) left over from an editing session.

ct () { function ct_usage () { comment USAGE ct oneFile~ }; [ $# -gt 1 ] && { ct_usage; return }; [ -f $1 ] || { ct_usage; return }; set – $1 ${1%~*}; trace_call $*; [ -f $2 ] || { ct_usage; return }; bkp $2; rm -f $1 } ctd () { [ -d $1 ] || return; pushd $1; foreach ct *~; popd; ctl } ctl () { find * -name '*~' | grep -v '\/\.'; }

The ct function is the workhorse. It takes advantage of foreach to run through all the tilde (~) files in a directory, i.e. those which have recently been edited. It backs up (bkp) it's ordinary non-tilde file and removes the tilde file. It leaves a warning message if either is missing.

The ctd function, armed with a directory-name argument, cleans up that directory, followed by a call to ctl which lists the remaining files for backup.

Today's commonplace clean-up fixed cblib, the ~ book library, where I audit the the text files, now just the word-count:

mkd_files () { egrep '\.(txt|mkd|src|m4|org)$' $*; }

Two problems need stating here in publishing to Leanpub.

1. after a few days off I have a hard time recalling the instructions to publish, given some new edits. Here's the sequence:
   1. leanpub.com, sign-in
   2. toolbar: your account, dashboard
   3. menu: books, under Published, edit
   4. menu: [=] preview, and you are where you need to be.
2. my Dropbox doesn't always sync. I think the latest is because of a Verizon reset. We'll see. Fix this first. Here's help: park the mouse pointer on the dropbox icon on the top menu bar. If all you see is the colored wheel for an extended period, restart the computer.

Today, after padding my txtFollow with possible duplicate MKD sources, e.g. m4, src, txt, mkd, .. I decided to learn a little about the ln command, and link up old, redundant copies to the current copy. And now, to avoid counting them, here's how I follow the txt:

uniqTxt () { txtFollow | xargs ls -i | printfirst | field 2; } txtFollow () { files commonplace html Family src shellfunctions \(* | mkd_files; } mkd_files () { egrep '\.(txt|mkd|src|m4|org)\)' $* | grep -v '/\.'; }

where uniqTxt has to get worked back into the counting functions.

tmpy () { fourspaces $* | fourspaces | tee ~/tmp/.y; } to_wlog () { yymmdd $(uniqTxt | textWords); }

and where I threw in tmpy as a reminder of how to collect code fragments for this document.

Some time ago, I left a note in my dbx.org "time for a sudo detector". Back on June 11, according to my records, I found this solution:

qsudo () { case $1 in sudo) shift; "$@" > /dev/null 2>&1 && { echo $1; comment you may need to clean up results of $@ } || echo sudo $1 ;; *) "$@" 2> /dev/null || sudo "$@" ;; esac }

It works for any command, e.g.:

qsudo mkdir -p a/fooey/path/to/nowhere

I just tested it.

Today I cleaned up the 'expn' library, balancing the credit union's credit card, and just reactivated my "plot" program for the word count in lib/word.log

I just realized is is redundant in markapp. So, after cleaning that one up, I'm into an audit of app\_fun which collects functions into an app. The jumping-off point is selfref, a generating function on the first argument which lists itself, and a collection of "like" names. Its purpose here is to list the function arguments which are the separate entry points into the application, since one function name does not encompass the whole set. Then, fun\_call, which, like app_uses, lists the functions called by the argument. The latter lists the first functions and it's called functions in a single list; the former lists them as a pair: caller, called. Looking at the tree command, not available on OS X, prompted that thought – to display the call graph. It turns out there is a tsort command, which produces an ordered list, with higher precedence first. Diving into my own awk call graph is too much work for the moment.

So, here are today's new functions:

sfuse () { set | fuse $1; } fun_call () { fun_uses $1 | awk "\$1 ~ ^\(1\\) { next }; { print \"$1\", \$1 }"; } mk_pkgapp () { echo mk_pkgapp mk_app mk_dirs mk_init mk_pkgname; } selfref () { eval "$1 () { echo $*; }"; }

Here, mk_pkgapp is an example of selfref in an _init call, and sfuse is an alias for the command line.

And, just for fun, the latest word-count plot:

80000 - |

70000 - |

60000 - |

50000 - |

*—|-----------------------------------------------------–—| 40000 - 500

If it looks like I'm padding the books, you're right! For the moment, the X axis is a crude count of days since Jan 1, 2013, and the y-axis is the result of a wc of the _markdown_able words in the directories under this search:

txtFollow () { files commonplace html Family src shellfunctions \(* | mkd_files; } mkd_files () { egrep '\.(mk|mkt|mkd|txt|src|m4|org)\)' $* | grep -v '/\.'; }

Two notes; I expanded the list to include org, m4, and mk for the inflationary period between 50 and 70 thousand words. And added a feature to reduce the bubble produced by an m4 -> src -> txt -> mkd production. (see the note for Tue, 24th). It's only partially implemented. I'll likely scale back the inflationary numbers when that book-keeping is fully implemented.

I mailed Bill and Jeffrey H about being late to deliver the updates to the book's appendix. I discovered Leanpub offers links to Dropbox/Public files. That changes (for the better) the layout of the appendix, shortening the text in behalf to better display the changes, in lieu of repeated blocks of code.

Which give me an idea; I just laid out a template serial edits. I'll take the time to re-use it in behalf of the changes I've made to the Factor Exercise.

More work, but better presentation.

Ah, yes. To support the templating, yesterday, I resurrected an old formatting tool – paragraphs – and refashioned the first, getpara, of a handful of paragraph managers. What is a paragraph? It's a member of a file of paragraphs, where the paragraphs are non-blank lines, inserted after a left-margin paragraph title, and delimited by a blank line. The getpara function accepts stdin or a named file following the paragraph title:

$ getpara paragraph [file]

as thus:

getpara () { trace_call $*; [ $# -lt 1 ] && { echo "Usage: getpara paragraph [file]"; return 1 }; [ $# -gt 1 ] && { [ -f $2 ] || { echo $2 is NOT a file; return 2 } }; awk -v para=$1 ' NF < 1 { printing = 0; } printing { sub("^ ",""); print $0; } { if ($1 == para) { printing = 1 } } ' $(shift; echo $*) }

The saga of the afternoon: in taking my suggestion above, I re-crafted the edits for the factor exercise. My original plan was to use diff3 to compare two successive files on the previous edit stream, and merge the discovered changes in the new, or current stream. What I've discovered is worth a chapter in a book. Let's if can keep this to a few paragraphs. This for much later, but a comparison of the potential MANPATH and the just-plain PATH:

/Applications/B1FreeArchiver.app/Contents/MacOS /Users/applemcg /Users/applemcg/Dropbox /Users/applemcg/Dropbox/git/bash-functions /Users/applemcg/Dropbox/rdb /usr /usr/local /usr/local/Cellar/ngrep/1.45 /usr/local/clamXav /usr/local/git /usr/local/lib/node_modules/npm /usr/local/lib/node_modules/npm/node_modules/request/node_modules/http-signature/node_modules/ctype /usr/local/texlive/2011/texmf/doc /usr/te

my problem arose in first using diff3, and after I'd found away around its not working, similar problems with sdiff:

… usage: diff -y … followed by a message of "acceptable" flags.

since this was happening with two clients of the basic diff program, a path search showed my diff was in /usr/local/bin. So, here's the fix. Into my HOME/bin/diff:

exec /usr/bin/diff $@

which guarantees running a conforming diff.

Now the rant. For a fun time:

man man

and search down the page for `SEARCH PATH FOR MANUAL PAGES`. I have contended for over 30 years, the answer to that question is the following:

manpath () { paths | sed s/\/bin$/\/man/; } paths () { eval echo \$${1:-PATH} | sed 's/::/:.:/' | tr : '\012'; }

you might dress up either by checking for the existence of the resulting directory name. Also, since this is a Mac OS X machine, hence a BSD-flavored Unix, the manual pages show up in needless `…/share/…' path component, factored out by these bits of magic:

find usr -type d -name man 2>/dev/null | tee ~.manpath comm <(cat ~/.manpath|sed 's/.man$//; s/.share//' | sort -u) <(paths | sed 's/.bin$//; s/\/$//'| sort -u)

which is what produced he bit above.

My point is, if manual pages were tied to their executable components, there is no way anyone could have gotten away with slipping a non-conforming diff program into my search, with out exposing its interface in a man page. What happened here is the `/usr/local/bin/diff` did NOT show off it's manual page. Reading the man diff was un-informative, since it was pointing at the conforming /usr/bin/diff, and NOT the diff found on the search path.

In one word: AARGHHH! Charlie Brown says as Yoda speaks.

And a last goodnight. After reading the introduction to Regular Expressions, I"m searching for the cummmings token RE. Let's try this for starters:

token: ([^ \(\),]|\[\(\), ])+

expression: ( *| {token} *| *{token} *\({expression}(,{expression})\) )

program: {expression}+

e.g.:

define ( +, args( a, b), plus (a,b))

define ( max, args ( a, b), if ( > (a, b), a, b) )

Now to go find interactive RE testers.

Here's the latest, after overnight thought on the cummings re's

escape: [\(),]

nonesc: [^\(),]

visible: [ tbsn]

token: ([^ (),]|\.)({nonesc}|\{escape}|\{visible})*

expr: ( ({token}( *\({expr}(,{expr})\))?)*)

program: ({expr})+

The open question of the moment: are parenthesis normal characters in a character-class?

And some cummings thoughts:

• symbols may be functions:

  define ( + , args(a, b), plus (a, b))

• tokens may be multi-word:

  define( subtract 2nd, args(a,b), subtract(a,b))

• functions may have variable arguments:

  define(max, args(a,ARGLIST, ), set(max, a), foreach(ARGLIST, if(greater(ARGLIST,max), set(max,ARGLIST))) return (max) ) comment( where ARGLIST may be any name, and is set in args and sequenced in foreach )

• implementation is a Tcl VM

• a comm function which guarantees "sort -u" file arguments
• r&d: a three-file comm, A, B, C, where columns 1-7
  1. A alone
  2. B alone
  3. C alone
  4. A & B
  5. B & C
  6. C & A
  7. A, B, & C

To build the commlib, collect the functions using comm:

sfuse comm | grep sort | field 1 fbdy $(sfuse comm | grep sort | field 1) >> commlib

Then pluck out the "sort"s and "sort -u"s from those functions.

Introduce the comm function:

comm () { function hasstdin () { case $* in -[123]*) ;; -) echo - ;; *) ;; esac }; function flagargs () { case $1 in -[123]*) echo $1 ;; *) ;; esac }; set – $(flagargs $*) $(hasstdin $*) $(shell_readable $*); trace_call $*; case $# in

1. report_unreadable $2 && return; case $1 in -[123]*) sort -u | command comm $1 - <(sort -u $2) ;; -) sort -u | command comm - <(sort -u $2) ;; *) report_unreadable $1 && return; command comm <(sort -u $1) <(sort -u $2) ;; esac

;;

1. report_unreadable $3 && return; case $1 in -[123]*) case $2 in -) sort -u | command comm $1 - <(sort -u $3) ;; *) report_unreadable $2 && return; command comm $1 <(sort -u $2) <(sort -u $3) ;; esac ;; ) trace_call $; comment USAGE "comm [-mn] file | - file" ;; esac

;; ) trace_call $; comment USAGE "comm [-mn] file | - file" ;; esac }

And put the other functions in the library back in their original libraries.

I'm now thinking of how to "dispose" of commlib; either in programlib or cmdlib.

tf: tt && fbdy

and now the startpage of the commonplace book is working:

file:///Users/applemcg/Dropbox/html/cook.html

visiting each of the links on that page, and following the first level of page links. I'm targeting there appearing in Dropbox/html, and testing locally.

Check out the thoughts of last month, Sat, June 7th.

Oh yes the book makefile sets the new standard. And a neat look-up:

find $(dotty html)/.. -name mk | xargs ls -lrt

Today in the IT Toolbox, I posted this note on using the [date command][itdatef]. The idea is that since so much logging is done with a date stamp, and we are talking about functions here, use of the formatting date arguement this way simplifies the logging . (e.g. no need for an "echo"):

mydate () { date "+%m%d-%T $*"; }

this offers at least two ways to log:

1. mydate log message … >> logfile
2. some cmmd | sed "s/^/$(mydate) /" >> logfile

and as the Mighty Dragon points out, there is a "just in time" issue with the use of a non-function. i.e don't put a statement like:

MYDATE=`date "+%m%d-%T`

at the beginning of a script, since the "date" has to be within any loop. Notice how functions take no notice of whether or not they are called within a loop, it's simply easier to invoke them when the behavior is required.

Today's functions, I'm dealing with reading my downloaded accounts:

• Wells Fargo
• Affinity FCU
• Fidelity, Cash and Investment accts.

The plan is like this: Positions are overwritten, Accounts are accumulated. Overwriting means just that. Over-write the file and record the contents in a relational data file, Accumulating means collecting the history and appending the latest data, sorting and trimming on the unique records. So, here's the plan:

At the moment, the only issue with the Portfolio Positions is the separate accounts are downloaded in like-named files; the account name is in the first field. Otherwise, the accounts identify themselves in the file name. So, in a sense, accumulating is easier than recording the history. A key function is the csv canonical collector:

cancsv () { sed ' s/^x / / s/," */,/g s/ *",/,/g s/^"/ s" *// s/, */,/g s/ *,/,/g s/ */ g s,n\/a/,/g ' $* }

The principal facets are:

• remove "'s (you can wonder what a " is used for, if commas don't do the job)
• remove spaces adjacent to ,'s

Special features (sort of peculiar to the data, but should be generic):

• trim n/a from fields
• trim a leading "x ", which I used to check off records I'd already recorded

So, any downloaded csv is "canonical" by a canoncsv and run thru this collection filter:

csvgather () { set – $1 $(basename $1|dateout); set – $1 data/$2 ar/$2; touch $2; ( cat $2; csvrec $1 ) | sort -u > $3; cat $3 | tee $2 } csvrec () { set – $1 ${2:-(prefrec $1)}; cancsv $1 | awk -F, "NF >= $2" }

Notice that csvgather collects an existing file, and then using csvrec gathers the existing, new files into the collection. The prefrec function runs thru the input file and returns the dominant field count indicating the records to collect.

A new function group **P**erson, **P**lace, or **T**hing, and a function to store a line of text for such a category; others are Idea, Test, …

Here:

• ppt – shorthand for pptlog
• ppt name – echos the log file name and **cat**s its contents to stderr
• ppt name and other stuff – puts "and other stuff" in the name log.
• pptsee name – echos the log file name
• pptl – lists all the ppt logfile names

• pptlog – lists the logged _name_s

  ppt () { case $# in

  1. pptlog 1>&2

  ;;

  1. set – $(pptsee $1); echo $1; cat $1 1>&2

  ;; ) date "+%Y%m%d %c : $(shift; echo $)" | tee -a $(home)/lib/ppt/$1.log ;; esac } pptl () { ${1:-echo} $(home)/lib/ppt/*.log; } pptlog () { indir $(home)/lib/ppt ls *.log | sed 's/.log$//'; } pptsee () { set – $1 $(pptlog); echo $(home)/lib/ppt/$1.log }

This group of functions reflects an idea I've had for about a year: a vanilla date function can also serve as a logging tool.

Here's a sample file from today's work:

/Users/applemcg/Dropbox/lib/ppt/idea.log 20140714 Mon Jul 14 13:33:19 2014 : for ppt, the Person, Place, Thing, … to remember 20140714 Mon Jul 14 13:49:09 2014 : If you're going to use a name, make it a function, not a constant. 20140714 Mon Jul 14 14:03:47 2014 : take shell functions on the road. find a training camp to host it. 20140714 Mon Jul 14 16:34:45 2014 : time to check the quotes of ..Diogenes Small..! 20140714 Mon Jul 14 16:53:05 2014 : time to start collecting pix in the commonplace book 20140714 Mon Jul 14 16:53:31 2014 : ideas can move to appropriate DOT org files 20140714 Mon Jul 14 16:54:03 2014 : find a way to mail/text myself a PPT! 20140714 Mon Jul 14 16:58:52 2014 : how about a PSV format for the ppt's

Simple, but a REMINDer comment in the TODO in the .org file

I often forget to log the latest version of functions I edit online. The three means are

• recall the command from the history and use the command-line editor
• use the ffun function to edit a few functions, and
• visit the lib where the functions is defined, and edit it directly there

I don't usually prefer the latter; two bookkeeping jobs are needed:

• after sufficient testing, record the functions in the log:

  $ dbsave library function …

• and after a number of modified functions have been collected in the library, it's time to consolidate the latest versions. As a quick aside, the approach I take relies on the shell's ability to remember the latest version of a function definition. A function may have many versions in a library. The last one in sequence is the effective version. So, to consolidate the library, I collect just the effective version of each function:

  $ group fix library

  Sparing you from group_fix for the moment, here are the functions I now use, timelyfun being today's addition:

  timelyfun () { fbdy $(T functions) | diff - $(T); } ffun () { gff fbdy $*; } dbsave () { trace_call $*; [ $# -lt 2 ] && { comment USAGE lib fun …; return }; [ -f $1 ] || { set – $(which $1) $*; trace_call $*; report_nonfile $1 && return }; chmod +w $1; fbdy $(shift; echo $*) | tee -a $1 $(T) }

  The timelyfun function compares the operative copy of the functions:

  fbdy $(T functions)

  to the copy recorded in the version history (in T), for today. Any differences mean it's time for another dbsave usage

Today, I cleaned up the finlib, adding functions to collect my current positions in the market. The method is build to extend to other CSV downloads, Wells Fargo and Affinity FCU. Here are the substantially changed functions:

collection () 
{ 
    trace_call $*;
    selecton $1 | tee $(needir data)/$1;
    indir data csv2rdput $1
}
csvrec	() { cancsv $* | awk -F, "NF >= $(prefsize $(basename $1))"; }
csvsnapshot () 
{ 
    trace_call $*;
    [[ -f $1 ]] || { 
	comment ERROR csvgather file -- no file $1, usually from $HOME/Downloads/*.csv;
	return
    };
    csvrec $* | collecton $(basename $1|dateout)
}
selecton () 
{ 
    trace_call $*;
    case $1 in 
	Portfolio_Position.*)
	    awk -F, '{ printf "%s,%s,%s\n", $1, $2, $4 }'
	;;
	*)
	    comment NO format FOR $1
	;;
    esac
}

The entry point now for market Positions is csvsnapshot. The way to call it goes something like this:

$ set – ~/Downloads/Portfolio_Position-*.csv # followed by $ ea # did I get the right files? $ csvsnapshot $* # and they update data/Portfolio_Position.csv $ …………. # and the data/h.Portfolio_Position.rdb.Z

The latter is a compressed file of the entry (and exit date) of any position downloaded from Fidelity, as well as any changes in position from purchase, sale, and dividends. To see the history for a symbol:

$ zcat data/h.Portfolio_Position.rdb.Z | row 'Symbol ~ AIG'

Given that I update daily.

The closing thought here is there will be a few functions which are sensitive to the data provider. Here you see selection is "format-smart". So is prefsize. When I repeat the drill for other accounts, a generic is likely to appear; I already have witnessed this for the "accumulation" accounts I mentioned previously. But the number of places is likely on the order of a handful. I'll re-visit them as they occur

This morning I thought it useful to work the rdbcmdlib into the financial mix, mostly to simplify fetching our investment position.

So, here are today's new functions:

acct_owner () { awk ' { c[$2]++; n[$2 "," $1] = $3 } function hold(i, o) { r=n[i "," o]; return ((r>0)? r: "") ; } END { print "symbol\tj\tm\tp" print "-----\t------\t------\t-–—" for(i in c) { printf "%s\t%s\t%s\t%s\n", i, hold(i, "j"), hold(i, "m"), hold(i, "p") } }' } acctid_sed () { column account key < acctid.rdb | awk ' NR > 2 { printf "s/%s/%s/\n", $1, $2 }' } cash_defl () { compute 'Symbol = ((Symbol != "")? Symbol: "cash")'; } fidPosn_rdb () { port_now | sed -f <(acctid_sed); } invest_acct () { row 'j+m+p'; } port_now () { zcat h.Portfolio_Position.rdb.Z | rdb_notime | cash_defl; } test_acct () { fidPosn_rdb | tail +3 | sort -k 2; }

The top in this case is test_acct. Here's a command line:

$ test_acct | acct_owner | row 'j+m+p' | justify

The little bit of magic here, since the downloaded data contains the Gr-kids 529k funds is the invest_acct function. The statement `row j+m+p` is the trick, it's an OR statement. At least one of our accounts __J__oint, __M__arty, or __P__at must have some money in it. And, in the interest of hiding syntax, it has it's own function. Replacing the command-line syntax with the function name removes the syntactic cuteness in favor of the meaningful mnemonic.

Note, the acct_owner function collects just that data, without regard to the account name. The `acctid.rdb` table has the key for the Fidelity account name, relative to my now single-character account holder. Our Grand-kids are similarly keyed, but not tallied here.

The acctid_sed function, as a command-line data file `<( … )` saves the trouble of storing a copy of the sed script which must be kept up to date with the rdb table. This seems to be worth the cost for the added generality.

I contributed some notes to the [lsof discussion][lsofdiscussion] on IT toolbox today. This is more a topic for either softwarereview or [shellfblog][] blogs. But, that's where I posted it.

For the moment, here's my update of scatter_page:

scatter_page () { set – scatter_man; set – $1 /tmp/$1.mkd /tmp/$1.html; source markapp; $1 > $2; markdown $2 > $3; open $3 }

I ended up promoting my [scatterlib][] functions, mainly the scatter awk program.

[filekeyfield]: http://goo.gl/91nXif "Split CSV based on column"

Yes, today for the first time, again from inspiration in ittoolbox, the original poster suggested the key for saving to separate files moved from the first to the third field. You can see quite a few responses, but I was moved to isolate the code fix to make the change to one field.

>_Thanks all for your responses, how would the process change if the Key were moved to the third position?_

Here's the thread to [split a CSV into files][filekeyfield] based on a column, where my responses are in the thread. But for now, the important development: using a function to return the name of the file to write (without saving the name as a variable). See the line with print $0 > filed($1,hdr):

kdcsv () 
{ 
    cat ${*:--} | awk -F, '

    function filed (n,hdr) {

        out = "FILE" n ".csv" 
        if (printed[n]++ < 1) { print hdr > out }
        return out
        }

    NR == 1 { hdr = $0; next }

            { print $0 > filed($1,hdr) }
    '
}
fun_files       () {
    awk '
     $2 ~ /\(\)/ && p[FILENAME]++ < 1 {
         print FILENAME 
     }' $(shell_onlyfiles)
}
xx () 
{ 
    . $1;
    fbdy $(functions $1);
    comment === $1 ===;
    read
}
logtool () 
{ 
    rdb_hdr function;
    functions itoolib | sort
}
fun_nit () 
{ 
    dbsave itoolib $*;
    logtool > itoolog.rdb;
    rdput itoolog.rdb
}
rdb_inserts () 
{ 
    set -- $1 h.$1.rdb.Z insert_time;
    [[ -f $2 ]] || { 
        comment USAGE rdb_inserts STEM of h.STEM.rdb.Z;
        ls h.*.rdb.Z 1>&2;
        return
    };
    shift;
    trace_call $# $*;
    rdb_hdr count $2;
    zcat $1 | column $2 | rd sort | rd uniq -c | sed 's/^ *//' | sp2tab | tail +3
}
ff_test () { foreach xx $(fun_files | sort -r) | tee -a .itoolib; }

The other great insight, to collect all the functions I'd been writing in the ittoolbox directory is the fun_files function. This can be used to discover functions in any directory which I've not already collected. The xx function is a throw-away, using the read in a foreach loop to watch the functions appear. The ff_test is another throw-away, just to demonstrate usage. For further development, turn the rdb_inserts into a more generic which will handle rdb_deletes as well. Look at the `set –` line for the clue to update.

And one last problem for another day; see ~/Dropbox/dbx.org, where it's time to distinguish between two types of RDB files (in my directories at least), where I have a command to collect all the data for a table, and those where some of the data is updated. The latter is the challenge, of course. This suggests a "table" of table, type, and mode. Later.

With this a.m's work, here are a few I've added to the `expn` directory. as I move the data handling for our expense record from Google docs, to Dropbox on this machine (and others).

913 transactions | column amount | grep , | wc

920 transactions | row 'amount ~ ,' | sed 's/,//' | tail +3 | tee account.fix

921 transactions | row 'amount !~ ,' | tee amount.ok

922 mv account.fix amount.fix

925 cat amount.ok amount.fix > transaction.rdb

Some of the amount fields had commas: 1,234.56. This is unacceptable for [rdb][] formats. So by line 920, I've figured out how to collect the amount fixes, mislabeling the fix file as `account.fix`. Line 921 selects the unaffected lines, and line 925 restores the transactions

Requirements for an expense entry function:

• accepts, date, amount, credit, debit, and comment fields
• default is today in YYYY-MM-DD format
• two arguments are amount, credit from owner's cash account
• an Environment variable: DEBIT, replaces a missing 3rd argument.
• three args: amount, credit, debit
• credit and debit accounts must be valid accounts; tested against account table and formerly recorded tax treatment is now a function of the account
• four args: amount, credit, debit and either date, or comment
• five args: date, amount, credit, debit, comment
• the comment may have spaces
• the results are returned to the user on stdout for storage, to be appended to an [rdb]
• formerly recorded tax treatment is now a function budget category.

The last week has been consumed by (maybe too much) work on my expn libraries. After an experiment on two levels, with a common library underneath, I've settled on one library. I've come to notice there are four distinct task areas; three are supported by functions in the booklib:

• manually entering expenses on the command line. see newxact
• after downloading a bank account, readying to reconcile with already-recorded expense entries
• manually editing the candidate pickbook file, eliminating the potential `book` records against `tran` records that didn't exactly line up, usually due to date mismatch, and
• programmatic loading those records into the transaction data

Today, I'm noticing the downloaded McCash - Entry.csv is a bit different from a bank download – the idea is to (see above) have a canonical format for the sources:

• Wells Fargo Checking
• Wells Fargo Visa Card
• Affinity Visa Card
• Affinity Savings
• Fidelity Cash Management

The booklib is a great improvement over the explib of yesterdays. Also, for fun, I've a libfunrdb. Here it is:

libfunrdb () { rdb_hdr library funcname; for l in *lib; do functions $l | sed "s/^/$l /"; done | sp2tab }

where today I needed to change the name of the field from function to funcname, since RDB using `awk` chokes on a field whose name is an awk built-in.

Today, I updated K&P's 1984 overwrite script to accept multiple file arguments. Here are the requirements and a sufficient demo/test

ovwn_dscr () { echo ovwn_{dscr,reqts,usage,demo,test,help,CMD}; } ovwn_reqts () { cat <<EOF

the requirement to overwrite an input file is completely separable from the other requirements of the problem. it was solved in '84 by Kernighan and pike for the single file case. it could/should be extended to multiple files. $ overwrite N file1 file2 … fileN command_to_run_toOverwrite args … here's a post (http://www.linuxmisc.com/12-unix-shell/3cb58927dbae8819.htm) extolling that now-30-yr-old breakthrough:

EOF

} ovwn_usage () { echo 'Usage: ovwn file … cmd [args]'; echo ' where CMD accepts ARGS followed by the first file … arguments,'; echo ' and then the same number of ovwn-specified output files.'; echo ' e.g. with THREE file arguments and ONE arg, the CMD must accept'; echo ' SEVEN arguments, first the arg, THREE inputs, and THREE output.'; echo ' the CMD must write the respective output file arguments.'; echo 'ovwn, having supplied the output files, then overwrites'; echo ' the respective input files preceding the CMD with'; echo ' the user-CMD-generated output.'; echo 'e.g. see ovwn_test, where XXX replaces "ovwn" with "overwrite'; echo ' in files foo and bar.' } ovwn_demo () { fbdy \((ovwn_dscr); echo '\) ovwn_test; set – bar foo ovwnlib; diff $1 $2; diff $2 $3' } ovwn_test () { cp ovwnlib foo; sed 's/usage/how/g' foo > bar; ovwn foo bar ovwn_CMD ovwn overwrite } ovwn_help () { functions ovwnlib; echo DEMO:; ovwn_demo | sed 's/^/ /' } ovwn_CMD () { trace_call $# $*; repl $1 $2 $3 > $5; repl $1 $2 $4 > $6 } $ ovwn_test; set – bar foo ovwnlib; diff $1 $2; diff $2 $3

After yesterday's outpouring, I'm fired up about the inspiration for an om set of functions, for *O**bject and **M**ethods. In the preceding collection, *ovwm is the name of the object and reqts, descr, usage, demo, test, help, and CMD are the names of the methods. The idea is that objects are the libraries, and a each will have a routine list of methods, supporting library maintenance and documentation, at least. We see some of those methods above.

The idea is to collect sets of methods by object, comparing objects by intersection, etc… of methods.

So, here's a list of functions with the most popular objects and methods:

bkp_copyright fun_starter profile_starter shell_init bkp_firsttime git_copyright program_copyright shell_qrf bkp_init git_firsttime program_firsttime shell_starter bkp_starter git_init program_init shell_test bkp_test git_starter program_list smartf_copyright cmd_copyright list_init program_starter smartf_doc cmd_firsttime list_qrf setpath_copyright smartf_firsttime cmd_init list_test setpath_doc smartf_init cmd_starter om_copyright setpath_firsttime smartf_qrf fun_copyright om_init setpath_init smartf_starter fun_firsttime om_list setpath_starter fun_init profile_doc shell_copyright fun_list profile_init shell_firsttime

The darnedest thing, using my favorite idiom: `set – $( … )`, I screwed it up for – oh – 15 minutes trying to unwind the effect of a `clear;` as the first command in the `set –`. DUH!?

File under "lessons learned", since the next command was to be `vi +/thing/ $1; shift`

For the record, here is the command turned into a function:

missinginit () { cat *lib | grep _init | field 1 | sort | uniq -c | grep '_init$' | awk '$1 < 2' | field 2 | grep -v '^~' | sed 's/_init/lib/' }

The purpose here is to find the *missinginit*s, those libraries who've lost their call to their own *lib*`_init` function.

Lest we forget, here is bkdb2thumb, which saves my Dropbox on a USB device:

bkdb2thumb () { ignore home pushd; set – Volumes/MJM_140818; set – $1 $1.dropboxcopy /tmp/bkdp2zip.txt; find * -newer $2 -type f | usefulfiles | cpio -pduvm $1 2> ~/lib/cpio.err | tee $3; date "+%Y%m%d%H%M.%S %c" | tee -a $2; comment $3; popd }

All of this is now on Dropbox and backed up.

And I just learned about `chflags` on Mac OS X. Some of the reason for bkdbfrerr is brought on by the immutable flag on a destination file. How they get set, I'll never know. And just when in Unix&Reg; history did that creep in. How does immutable differ from the `read only` attribute? Anyway bkdbfrerr, followed by the `chflags` call to clear the hurdle:

bkdbfrerr () { home cd; set – $(grep 'Operation not supported' ~/lib/cpio.err | sed 's/.*cpio: /; s:.*//'); dir=/Volumes/MJM_140818; ls $* | cpio -pdvm $dir; echo $# $* } $ chflags -v nouchg /Volumes/MJM_140818/html/*.html

Katie's wedding intevened. Wonderful event.

But now its' time to put in the UTC/GMT clock. I enrolled for the MIT/Harvard [edX Intro to Computer Science][edxics], in Python. Since all schedules are in UTC, I thought it would be good to keep handy.

See the top of the page.

Big developments yesterday

• good work on reconciling family expenses with accounts, using command-line functions rather than spreadsheets, and novel

• a generic – i.e. low-level function help, s.t. any function may employ user help. So, for example:

  `$ foo help` may be useful.

  what this does is encourage each function to have an early call:

  … help_msg $*

  which is benign for all arguments except `help …`. From the function foo, called `foo help` will invoke an existing `foo_help $*` or announce there isn't such a function.

Here is help_msg:

help_msg () 
{ 
    [[ $1 == "help" ]] || return;
    shift;
    set -- ${FUNCNAME[1]}_help $*;
    trace_call $# $*;
    declare -f $1 > /dev/null || { 
	comment NO help for ${FUNCNAME[1]}, need a function $1;
	return
    };
    doit $*
    }

The magic here is supplied by the `${FUNCNAME[1]}` variable which returns the name of the calling (up one level on the call stack) function.

Theses functions highlight my progress on our account balancing:

tbal () { set – ${1:-fid:cash}; cat transaction.rdb | balance $1 } grunBal () { set – \(1 2014\){2:-0400}; tbal $1 | row "date >= ${2}" | compute ' \ comment = ((comment ~ ^$ && credit == debit)? \ "running balance": comment)' | rd grep running | compute 'balance -= amount' | ncolumn debit }

where `grunBal` is dressed up to better show its behavior:

1. retrieve the transaction balances for an account such as `fid:cash` after a date defaulting to Apr 1, 2014, and
2. compute a `comment` which if empty and the `credit` equals the `debit` field, which is my indication the `amount` field carries a bank-supplied account balance, or defaults to running balance since I now have a command to quickly supply any day-end balance, from a bank listing, then
3. pick out any line with running (`rd grep running`), and
4. recalculate the `balance` by subtracting the `amount` entry, so, when displayed, a balance entry of 0 shows the trial and stated balance agree for that date.
5. the `debit` column is removed, since it is largely redundant and the space is better used by other fields on the terminal display.

I've just dug back in to the Python course. Time to get busy since the current exercises are due by 23.30 UTC tomorrow. Which is but 30 hours from now.

It's now going on 11 pm, and I've finished lecture 5 and 3 or so exercises in lecture 6. I'll finish with time to spare..

if I wake up. C U in the morning.

It was a real scamper. I was working the problem by 8 a.m. and making good progress on the exercises. But then there was this closing problem set for the weekly lecture. It was due at 23.30 UTC; it's now 20.00 on that clock. I finished a few hours ago, but really struggled with the problem set: play a game of hangman, interactively with a user.

The interaction was easy enough: raw_input is the key there. A simple list of paying attention to detail bit me:

• Pay very close attention to the details of the print statements, and
• Pay even closer attention to printing the status. I'd left out a characteristic character-based horizontal rule separating each step and the outcome: success or failure.

It was a fun exercise; oh yes, the other mistake. I solved the whole thing before reading the intermediate exercises to solve three helper functions. Had I done things in the right order, I'd surely have saved a hour or better. Looking at the helpers though, they didn't seem to fit my view of the problem. The big hang-up was in preparing the `display` variable which showed the results of the guesses so far:

m a _ _ _ m _ _ _ w a _

for 'martymcgowan'. A one-character occurrence was trivial. I was off by one for multi-char letters. The helper function would have helped here. The insight was not to keep a permanent copy on each step, but re-calculate on each successful addition.

Week 4 exercises can now proceed more leisurely.

I thought of this idea while waking this a.m: how about a job to back-up changes to `.rdb` files, where I'm now storing our financial, expense data.

Here they are:

rd work () { while true; do foreach rdbkup *.rdb; sleep 600; done; rm -f test.err } rdbkup () { set – ${1%.rdb}; newest h.$1.rdb.Z $1.rdb || logjob $(doit rdput $1.rdb 2>&1) } safeAppend () { file=$1; shift; chmod +w $file; "$@" >> $file; chmod -w $file }

The basic idea is to backup changed rdbfiles when they are newer than their history file. rdput does the heavy lifting, adding two fields to every record: `insert_time` and `delete_time`. If a record appears to be new, it's inserted. If it's not in the current table, it's been deleted. These are recorded in the `h.$1.rdb` file, and that is compressed into the `h$1.rdb.Z`

The logjob function uses safeAppend to record its arguments after a leading timestamp in the local `joblog` file.

Oh, yes. This was born from the idea of splitting off changes to any `.rdb` file into a separate file and then replacing them in the original file. Here's the plan:

1. from a file.rdb, query the records to be fixed to a file.fix
2. diff those two files, leaving the original records in a file.tbl
3. update the file.fix records with an editor
4. append the updated records to the file.tbl and return to file.rdb

Today I added a few functions to remind me which functions I should be saving, and for sure, learning. A brief prospectus:

• `recentFunctions` – in the last 120 commands, which are more than one character or are not a positional parameter: $N.
• `learnMe` – which recent functions are on the list of ignore-able function names
• `lgnoreable` – some of the names turning up on the list to learn are ignore-able. Add them to the list.

The Code:

ignoreable () { foreach echo $* | grep -v '^.$' | tee -a ~/lib/ignoreFunNames.txt } learnMe () { set – ~/lib/ignoreFunNames.txt; comm -13 $1 <(recentFunctions) } recentFunctions () { th 120 | nhn | field 1 | egrep -v '^(.\(|\\))' | sort -u }

A function I've had a long time in the offing, `fun_files` reports the file names used in a function. Notice that `learnMe` and `ignoreable` both use the same file. The latter accumulates ignorealbe function (and command) names; I know them well enough without a helper. And the former lists the `recentFunctions` which are not on the ignoreable list, and either in need of more practice or a help function.

Today's top-level function is `fun_open`. Using `ffun`, the function editor whose arguments are functions to fix, and `dbsave` which appends functions to library, then `fun_open` lists just those functions whose latest edit occurred after any recent save.

And `fun_files` produces the list of named files in a function. We start with an example of its use and the function bodies I've just mentioned.

$ fun_files ignoreable /Users/applemcg/lib/ignoreFunNames.txt = = = = = = = = = =

fun_files () { foreach namedfile $( fbdy $1 | tokens | sort -u | awk 'length(\(1) > 1' ) } tokens () { sed "s/[;\(\)={}']/ & /g" | tpl; } fun_open () { ngh ffun dbsave | grep -v '\\)' | nhn | uniq | awk ' $1 ~ ^ffun$ { for (i=2; i<=NF; i++) open[$i] = 1 } $1 ~ ^dbsave { for (i=3; i<=NF; i++) open[$i] = 0 } END { for (o in open) if(open[o]) print o } ' }

Here `tokens`, taking some notice of shell syntax produces a token per line using another function `tpl`.

are done! So here's today's good stuff: The challenge is to see on what dates did I do some entry in any database. rdb_bydate is the top of the chart here. Here is it' most recent report on our `transaction`s:

Count insert_time –— ------–— 893 140806 4 140807 7 140808 10 140809 27 140814 3 140815 13 140819 18 140820 22 140821 4 140823 5 140825 10 140903 10 140906 73 140907 5 140909 3 140910 12 140918 16 140921 10 140925

which says that on August 6, I entered 893 records, … and yesterday I added 10. The first entry is the initial download from a Google spreadsheet. Subsequent are manual entries, some are downloads from our accounts. There are some fresh tools needing reporting on how I've encouraged our account balancing.

In any case, here the rdb_data will be a common prefix to any `rdb` file I'll be working with. rdb_bydate sorts a column of the existing record's *insert_time*s, saves the YYMMDD from the time, and produces a population by date. N.B. rdb_data could save time by using newest between the table and its history. Also rdb_whinsert should deliver the EMPTY rdb file on standard out.

rdb_whinsert () 
{ 
    set -- h.$(rdb_data $1).Z;
    [[ -f $1 ]] || return;
    zcat $1 | row '!delete_time' | ncolumn delete_time
}
rdb_data () 
{ 
    set -- ${1%.rdb};
    set -- $1.rdb;
    trace_call $# $*;
    [[ -f $1 ]] || { 
	comment no RDB file $1;
	comment local RDB;
	ls *.rdb 1>&2;
	return 1
    };
    rdput $1;
    echo $1
}
rdb_bydate	() { rdb_whinsert $1 |
    column insert_time | rd sort | sed '
	s/14\([0-9][0-9][0-3][0-9]\)\(......\)/14\1/
	' | rduniq
}

Finally, with some work… the newer concept is dotlib.

On entering a directory by `cd, pushd, …` you can now routinely

$ dotlib

since if there is a file `.dotlib`, it will be 'source'd. What goes in that file? Any library names that need their functions available, and … this just in today: it's the place for local functions. Here is the `.dotlib` in my `Family/finance/data`:

source finlib pp4f () { echo Account Symbol Quantity Cost; } spick () { pick $* >> .pick.out; cat .pick.out } scol () { column $(echo \(* | tee -a .column); } srow () { row "\)(echo \"$1\" | tee -a .row)"; } quanbysybol () { function _foldtoposn () { trace_call $# $*; rdb_hdr Symbol j m p; tawk ' BEGIN { F[0] = "j"; F[1] = "m"; F[2] = "p" } NR > 2 { s[$2] = 1 p[$1] = 1 q[$1,$2] = $3 } END { for (symbol in s) { printf "%s", symbol for (i=0; i < 3; i++) { a = q[F[i],symbol] printf "\t%s", a } printf "\n" } } ' }; trace_call $# $*; ignore needir .tmp; cat PortfolioPosition.rdb | column $(pp4f) > .tmp/posn; rdjoin .tmp/posn acctid.rdb | scol account Symbol Quantity Cost | _foldtoposn | row 'j+p+m' }

A little exposure to details here, the columns `j, p,` and `m` show **J**oint, **P**at's, and **M**arty's position (i.e. number of shares of any instrument).

The `.row` and `.column` files collect the arguments to the row and column rdb commands used in the local directory.

And yesterday, I moved my cummings project out of this Dropbox and back to GitHub. Getting comfortable with the git functions was not that bad a challenge.

At the risk of getting side-tracked on other important stuff I need to describe the newly-evolving function update sequence.

It all starts with dotlib. Recall how that function assumes a `.dotrc` file in the local directory. It started as a place to "source", or include the contents of useful libraries. I've since included adding a PATH component in the .dotrc file. And most recent, I'm now including function needing to be updated.

Here's the sequence:

1. A function exhibits questionable, or poorly understood behavior, so
2. look at its fbdy,
3. look at the body for other functions, and include them in the search.
4. having identified a function or functions needing closer inspection, then append their body to the current .dotrc and edit in the changes or diagnostic information
5. source the updated .dotrc: `$ dotlib`
6. test, correct and use until satisfied. Now here comes the fun part.
7. identify the source library where the function lives: `whf function`

8. find find and update functions in common with the library:

   $ xx () { comm -${1:-12} \ <(functions $(which ..lib)) \ <(functions .dotrc) } $ xx # "xx" is a throw-away function name $ # if satisfied, collect the function names to update $ set – $( xx ) $ dbsave …lib $* # updates the lib, now clean out .dotrc $ fbdy $( xx -13 ) > .y $ fun_starter dot >> .y $ mv .y .dotrc $ unset xx

And in a just-reached decision, I either need to update funclocn, which finds *func*tion *loc*atio*n*s, to add a local `.dotlib`, or, since funclocn already looks for a local `./.\*rc` file, I can change the name of the dotlib target to be some rc file. In the spirit of no flash cuts, I can move or link selective .dotlib's to say `.dotrc`'s

I'll do that now in one of the directories with this feature.

I've updated the `.dotlib`s to `.dotrc` and it's working just fine.

I've moved the discussion and code to its own document

Today I discover the FODS principle. I've been doing this all along; today I formalized the practice in F**unctions **O**f the **D**ay **S. Note slightly out-of-place 'S'.

In any case, here are todays:

best_balance () 
{ 
    set -- $1 ${2:-20140000} ${3:-1.00};
    trace_call $# $*;
    long_balance $1 | row "date > $2" | rd sort -r | long_within $3
}
cleandot () 
{ 
    set -- $(which ${1:-booklib}) $2;
    [[ -f $1 ]] || { 
	comment NO library: $1;
	return
    };
    comm $2 <(functions $1) <(functions .dotrc)
}
long_within () 
{ 
    set -- ${1:-1.00};
    trace_call $# $*;
    row "long < $1 && long > -$1"
}

Two fall back in the booklib, while cleandot, with a booklib bias shows how I'm recovering the recently repaired functions. Lately, `.dotrc` is where the fixing is taking place. This seems to be more reliable than trusting to the memory of the current shell's environment. That's why God invented files: to serve as off-page memory locations. So, for the time-being, my local memory is captured in the `.dotrc` file. So, cleandot, using comm's set-arithmetic facility:

• -12 shows the common members
• -13 shows the set-subtraction of the first from the second,
• -23 – vice versa

making it possible to simply restore functions being updated in `.dotrc` to their original library:

1. cleandot {lib} -12 # which functions in .dotrc belong to {lib}, so
2. dbsave {lib} $(cleandot {lib} -12) # updates {lib}
3. fbdy $(cleandot {lib} -13) > .l # copies other fbdy's to .l
4. mv .l .dotrc # and restores to the library
5. fun_starter dot >> .dotrc # so it behaves like a standard library

This process may be repeated for any libraries whose functions may have been pulled into `.dotrc` for inspection and repair. The list of libraries whose functions are being repaired in .dotrc is available by:

$ foreach do_whf $(functions .dotrc)

which displays the function names, and the source library or libraries.

Speaking of Function of the Day, I've overlooked fods itself, and its companion spick. But the winner is probably 3, 4 days old: nf, a shorthand for awk's NF parameter. Here, the nf command is uses as a keyboard short-cut as a query, for example:

$ … | nf lt 5 # prints lines with less than 5 (space-separated) fields

and in case you need CSV or TSV, I've a couple of short-hands laying around which account for that.

$ … | nf cawk # prints the number of fields followed by the CSV record

This was invented to do a quick audit of CSV, TSV files where you should be expecting a uniform number of fields in each record.

here are today's FODS:

fods () { spick $(T functions|sort -u); } nf () { case $1 in *awk) AWK=$1; shift ;; *) AWK=gawk ;; esac; case $1 in lt) $AWK "NF < ${2:-5}" ;; gt) $AWK "NF > ${2:-1}" ;; ne) $AWK "NF != ${2:-5}" ;; *) $AWK '{ print NF, $0}' ;; esac } spick () { pick $* | tee .pick.out; }

If ever there were a FOD, it's fun_changes. or when did the function have it's latest change?

Here's a sample output on funlib:

fun_changes 20141013_Mon_04:50:21 fun_stamps 20141013_Mon_04:43:04 whf_open uniqTxt txtFollow txtDups tokens

note, not all functions will be tagged; they were last modified before *fun_changes was available. Now it's time to make an RDB table. But first, here are the relevant function bodies:

dbsave () { trace_call $*; … chmod +w $1; ( fun_timestamp $(lib_handle $1); fbdy $(shift; echo $*) ) | tee -a $1 \((T) } fun_timestamp () { date +"\){1}_timestamp () { echo %Y%m%d_%a_%T%t.; }"; } fun_stamps () { trace_call $*; cat ${*:–} | awk ' $1 ~ ^[a-zA-Z0-9_-]*_timestamp$ && $2 ~ ^\(\)$ { stamp = $5; next } $1 ~ ^[a-zA-Z0-9_-]*$ && $2 ~ ^\(\)$ { printf "%s\t%s\n", $1, stamp } ' } fun_changes () { fun_stamps $* | sort -r -k2 | printfirst; }

fun_changes reads function libraries for **fun_timestamp**s, printing the latest occurrence of any functions updated (via dbsave) in their respective function library. The lib_handle function returns the basename of a function library, stripped of any leading dot (e.g `.profile`) and trailing `lib` or `rc`. e.g. `~/dropbox/bin/cmdlib` returns `cmd`. This produces a function named cmd_timestamp, which is appended to the function library just before any other functions being added.

And, just for drill, here is the first copy of the function history, using the functions which created it:

insert_time delete_time library funcname yymmdd dow time ------–— ------–— --–— ---–— -–— — -— 141013063237 profile allf 20141013 042934 Mon 141013063237 funlib fun_changes 20141013 045021 Mon; 141013063237 funlib fun_hdr 20141013 055033 Mon 141013063237 funlib fun_rdb 20141013 055033 Mon 141013063237 funlib fun_stamps 20141013 055033 Mon 141013063237 funlib fun_timestamp 20141013 055033 Mon 141013063237 funlib fun_current 20141013 061723 Mon 141013063237 funlib fun_tordb 20141013 062702 Mon 141013063538 funlib fun_history 20141013 063259 Mon

Oops. Note the header names on the table. Have to swap `dow` and `time`.

After some time with th python course, and some serious and important family events, it's time for an update. A few recent FODs:

fun_setstamp /Users/applemcg/git/bash/bin/funlib size_comment ./ourbooklib accounts Users/applemcg/dropbox/bin/booklib tsf ./dotlib ..dotrc qrecorded ./.dotrc fun_diff /Users/applemcg/git/bash/bin/funlib

fun_setstamp	() { date +"${1}_timestamp () { echo %Y%m%d%t%H%M%S%t%a%t.; }"; }
size_comment	() { trim_tran_comment | compute "comment = substr(comment, 1, 24)"; }
accounts () 
{ 
    trace_call $# $*;
    rdb_history account | rdb_current | rd sort
}
tsf () 
{ 
    set -- ${1:-$(expr $LINES - 4)};
    tail -$1
}
qrecorded	() { transactions | rd grep $1; }
fun_diff () 
{ 
    f=$1;
    set -- $(whf $f);
    [[ $# -gt 2 ]] && { 
	set -- $(pick $*)
    };
    f2file $1;
    comment '< fbdy';
    comment '> library';
    fbdy $f | diff - .tmp/$f
}

The first thing to note, from the `foreach do_whf $(< .y)` report is three of the six functions selected are only available locally, two in the `.dotrc` file and one in `ourbooklib`. So, here's the evolving process:

1. the `.dotrc` and its `dotlib` link is for diagnosing and working on functions already stored in the public libraries.
2. its also the place where locally generated functions should go first
3. then after some maturity, they may be moved to a local library, in this case `ourbooklib` indicating a future home in `booklib`.

However, (a big one) this process points to the ability to not put everything in a public library. Local functions may remain local. This is the great benefit from the dotlib approach.

For the FOD's: fun_diff was updated to a different model than previous for two reasons. First, for simplicity, and it turns out this had a tremendous ( 3:1) performance improvement over the former method of dropping into a lower shell, sourcing the library, and capturing the function from that environment onto the output in the the current shell. This approach, using f2file captures the function bodies from a library, without `source`ing the library, one per function in files in the `.tmp` directory. The current useful function body is then compared to the function reported out from its home library. Not the use of the `pick` command, should the `whf` (where's the function) query turn up more than one library. This whole function might be further wrapped to avoid gratuitous updates of functions copied into the diagnostic .dotrc, but not updated.

tsf is the mnemonic for **T**ail a **S**creen**F**ull

accounts, along with categories and transactions shows the now-standard way of reporting out those tables in the family expense record

Oh yes! The other day I learned the built-in `pushd`, with no arguments behaves as my (now former function) `swapd`, which I've pulled from the repertoire. Maybe I'll need an alias to remind me it's now gone. Which pushes an idea on the stack: how about a general facility, using the alias. e.g. instead of indir being a function passing it's args to the "official" shell_indir, it may as well simply be an alias, and not intrude in the function name-space. Oh well. Just and idea.

now for the portable functions, ksh and bash

I've recorded a note in `commponplace/unixshell` on the possibility of writing functions portable between ksh and bash. Here's the opening practice. Identify the low-level facilities to build up. I think this is the starting list: type, functions, command and alias. Here are the first entries in a portable library:

isKsh() { one=1; [ one -eq 1 ] 2>/dev/null; } th() { set – \({1:-\)(expr ${LINES} - 3)} isKsh && { history -$1 } || { history | tail -$1 } } fbdy () { isKsh && { functions ${*:-fbdy} } || { foreach f_fnb ${*:-fbdy} } }

Here isKsh is the way Dave Korn recommended to me to test for being in the ksh or not. So, two user functions which evaluate the state of the running shell at execution time. The alternate approach would be to define a set of functions in one place when logging in and setting your environment. I currently use:

$ . ~/.myrc # or, its alias: $ itsme

The point here is, looking at the th ( **t**ail **h**istory) function.

1. the set built-in sets the first positional argument to either the first supplied argument or a default value of the number of lines on the user's screens less three. This line is works identically in both ksh and bash.
2. this line tests to see if ksh running, and if so, executes the ksh command to return a number of history lines, since in ksh, the history command takes the number as a flag argument.
3. alternatively, if you are not in the ksh, but bash the final line `|| { history …` says "or, if not, then" and executes the bash version, then uses the tail command to deliver the requested number of lines.

This alternative approach is taken because the two different history commands are incompatible.

The fbdy example is similar, but different, since the two shells use different means to retrieve the functions from the environment. In ksh, functions is an alias for `typeset -f` and in bash, at bottom, fbdy uses `declare -f`. At the moment, I've written functions to retrieve the function names from a library whose definitions have been filtered thru fbdy, which has the facility of preserving one-liners in one line of code.

What this tells me is I should re code my functions in another name, and make it more robust, since it relies on the bash practice of returning a beautified version of the code rather than the code as entered, which is the practice in ksh.

And, before I forget. I will have to hide most of the bash syntax from the ksh version, since there may be some syntactic difficulties to great to allow them to be _source_d in either envenomed. For example, in fbdy the f_fnb function hides some work to restore the one-liners as one-liners, since `declare -f` produces a four-line format.

From Houston. I'm learning the big thing I need todo is put all the ~/git directories on [github][]. I know I'm missing bash and eec. Dropbox/git was "simpler" in a sense from a "let's edit it here" sense, but I like the git hurdle since it encourages the regularity of a 'commit', especially for code.

The current exercise is out of [backash][], and that is turning each shell library into a ksh or bash library. Each library may be "source"d in either shell. The requirements are in the document.

The good lesson from the backash project is in discovering how many places I've embedded bash function name identification: e.g. `$2 ~ /()/`

I'll start by unwinding a few of those: gh, fun_uses, … as the need arises.

I reclaimed fun_names for app_fun*, which will take later unwinding and in shlib's, renamed to fun_from, a better name since, the question is "what are the fun_ctions _from these libraries?"

Based on a `foreach do_whf ` functions matching ^fun\ not in funlib_, we learn that whf will have to be canon smart. What is "canon smart"? It will be some time before my function libraries are all of the form:

function name { … ; } # the CANONical form, rather than name () { … ; } # the default form in bash, declare -f …

Therefore, "canon smart" is a requirement on a function, looking for function names which match either format.

And, why is that?

Not every function can be blithely converted to the canonical format it an otherwise smart library. The smart library is one which may be `source`d in either shell.

Yesterday's lesson on tmp files. The `ncolumn` function was giving me fits. It used a temporary file `$TMP/.input.$$` to hold the standard input. The need arises since, in order to trim column names from a file, the purpose of ncolumn, it's necessary to collect the current names to produce o list of those remaining after pulling out the list to be removed.

When a tool like ncolumn is a shell file, it's a separate executable, and the value of the `\[` to the shell is the process id. So, in a sequence of piped commands, possibly not all on the same command line, two or more instances of the ncolumn function are writing to the same file. The solution is `$RANDOM` in place of `\]`. And since there's a 1:10,000,000 chance of collision, I think a function random_tmp_file is in order:

function random_tmp_file 

    set -- tmp/${1:-.seed}

    while true;
      do f=$1.$RANDOM
	 [[ -f $f ]] || {
	    echo $f
	    return
	 }
     done

dotlib: the problem arises from the above discussion. today i took down booklib and collected it's functions in the local ourbooklib. In the current scheme, the profusion of places to look and fix had grown out of control. So, now, as a function matures, it may start in `.dotrc`, sourced by dotlib, it may move on to some _local_lib, where it should live for some time, until its general nature becomes apparent to some other library, at which time it may be promoted to one of the …/bin/_general_**lib**s

I've added the [development notes for cummings][devnotes]

Now that I'm working with tw5, and these notes, I need to put the products in an accessible location. First, I've cranked up [MAMP][], as [localserver][] ultimately to test my word press on my home machine in preparation for producing on the web. Hey wait a minute. That makes three means of document preparation:

• html via markdown
• Wordpress using php, and
• TiddlyWiki using javascript, though tw5 advertises markdown features

Ok. so what do sleepers and toFrom do for us?

toFrom takes two file arguments and occasionally links the first to the second, when the first changes. a third argument supplies an interval to sleep to keep checking for updates.

then sleepers displays the PID of the sleeping commands, so that this command kills the sleepers:

$ kill -o $(sleepers)

Here is their code: p

toFrom () 
{ 
    [[ -f $1 ]] || { 
	comment "USAGE toFrom file dest [N]";
	return 1
    };
    set -- $1 $2 $(expr 1 + ${3:-4} \* 12);
    while true; do
	newest $2 $1 || ln -f $1 $2;
	sleep $3;
    done &
}
sleepers	() { ps -ef | awk '$(NF - 1) ~ /sleep/ { print $3 }'; }

Notice, in toFrom, the default value of the third argument is "4", and it's part of an expression to calculate the sleep interval. The idea is to calculate relatively prime intervals for the sleeping processes.

Here's a sweet little bit: a TV Jumble (Newark Star Ledger) puzzle solver.

For the uninitiated, the TV Jumble has four scrambled words, today's being `KAYCNR, SMOTECU, NOVTMER`, and `MARHBEC`. Each is accompanied by a clue: "Archie could be this, …". To solve the puzzle, unscramble these words into a grid with a letter per, where some letters are circled. For the first word, the third and sixth are circled. It turns out the first word is "CRANKY", therefore the A and Y are circled. You then collect these circled letters from the four jumbles to solve the whole puzzle whose clue points to a TV show or personality.

These two functions do the job. The heavy lifting is done by grepN, which constructs a sequence of `grep commands to filter a word list. I've an 86000 word list, left over from my recent Python course, which is useful enough. The tvJumble function defends both against misuse. I now try to write a helper message into each function so that merely typing its name offers the necessary clue.

tvJumble () 
{ 
    [[ $# -lt 1 ]] && { 
	comment USAGE tvJumble DROW [wordlist ... ];
	return 1
    };
    word=$1;
    shift;
    comment searching $word $*;
    cat ${*:--} | eval $(grepN $word)
}
grepN () 
{ 
    quietly comment give a word, construct a series of greps;
    echo $1 | awk '
	function grep1( l) {
	    if (g[l]++) return
	    printf "%sgrep -i %s ", p, l; p = "| "
	}
	{ for (i=1; i <=length($1); i++) {
	    grep1( substr($1,i,1))
	}
	print ""
    }
    '
}

To appreciate how it works, here's `grepN` at work:

diary.$ grepN KYACNR
grep -i K | grep -i Y | grep -i A | grep -i C | grep -i N | grep -i R 
diary.$ 

And here's an instance of using `tvJumble`:

py.$ tvJumble KAYCNR ProblemSet4/words.txt
searching KAYCNR ProblemSet4/words.txt
CRANKILY
CRANKLY
CRANKY
CRYOBANK
KNACKERY
py.$ 

A future enhancement might trim the list for the length of the word. Plurals are the common extraneous result.