Simon icon Simon
Flexible server monitoring

New Filter Variable: PreviousFilterPreviousText

Currently we have "FilterPreviousText", which gives the output of that same filter from when it was last called. Because it is reflexive (looks only at itself), it is impossible to leverage it for any kind of comparative analysis of output, even when a new filter is scripted.

If we could have a new variable that looks at the output of the *previous* filter from when it was last called, then this would open a whole world of comparative analysis possibilities. If no previous filter exists, it could default to the existing behavior of "FilterPreviousText".

What do you think?

David Sinclair's picture

Re: New Filter Variable: PreviousFilterPreviousText

If I understand what you mean, I think you can already do this.

If you have multiple filters, you can insert a digit after the "Filter" prefix to refer to variables from an earlier filter.

For example, if you have two filters, the second one can refer to the {FilterPreviousText} of the first via {Filter1PreviousText}.

Re: New Filter Variable: PreviousFilterPreviousText

Thank you for the prompt reply David. I would have missed it if I hadn't checked back here - is there an option to have notification by email of replies to a thread I start/participate in? [EDIT: I've added a test in Simon to alert me].

From your reply it seems that you do indeed understand what I mean, however I can't get your suggestion to work. I haven't tried it in a bash script yet, but using {Filter2PreviousText} in a format filter (3rd or subsequent filter of course) fails to function as a variable - it just returns the string "{Filter2PreviousText}" just as it would with any other non-variable.

I'm encouraged that you seem to have a feeling you've implemented this already though, and I'm hoping that you can help me access it.

I really like using Simon btw - I'm glad it exists :)

David Sinclair's picture

Re: New Filter Variable: PreviousFilterPreviousText

Unfortunately my forum software doesn't have an option to email notifications of followups. That's something I'd like to add one day. But yes, using Simon is a perfect solution. I use Simon for this myself (looking at the Recent Posts page, to see any updates). There's actually a built-in test for this, though I realize you might want something more focused.

You're right, it looks like {FilterPreviousText} is a special case, and doesn't seem to support the feature of referring to previous filter values, ironically. I've spent a bit of time exploring the code, and haven't found exactly why, but it does work a bit differently than other variables. I'll make a note to explore some more sometime, and see if I can make it work somehow.

Sorry about that. In the meantime, it may be feasible to do the same thing by using {FilterPreviousText} as part of a script.

I'm glad you like Simon!

Re: New Filter Variable: PreviousFilterPreviousText

The trouble with using it in a script is much the same as the trouble with using it in any other kind of filter, in that it can only return its own previous output - i.e. if "X" is the previous output then using {FilterPreviousText} on it's own means that "X" is the only possible output ad infinitum.

If the current output from a higher filter is added by way of a variable such as {FilterInputText}, let's call that output "Y", then that gives iterations of:

Y
Y Y1
Y Y1 Y2
Y Y1 Y2 Y3 (and so on)

The output just gets longer and longer as iterations increase, making any comparative analysis of Y and older Y messy and impractical to keep running.

For the time being, as a workaround, I've implemented the following two bash scripts:

## take previous output and append the separation token "_" between it and the new output from a higher filter and, in the process, remove the OLD old output by looking for the first separation token in the string.

old="{FilterPreviousText}";
echo -n "${old#*_}_{FilterInputText}"

That yields:

_Y
Y_Y1
Y1_Y2
Y2_Y3 and so on.

which I use in conjunction with the following script:

## Returns elements unique to the new list when compared to the old one, where newlines separate items in the list

## A bug (apparently undefeatable) causes the last line of the new list to be duplicated, which messes up the the comm, so the new list is separated before the comm, and the last (duplicate) entry is removed in the comm.

old_new="{FilterInputText}";
new="${old_new#*_}"
comm -13 <(echo "${old_new%_*}" | sort) <(echo "${new%$'\n*'}" | sort)

This yields elements of the new output that were not present in the old output, which is extremely useful to me and pretty much what I'm looking to accomplish. It's not ideal though because the "shadow" of any change appears in the filter the next time the test is run and yields an empty change, the possibility of which has to be looked for and dismissed in following filters.

Incidentally, the bug I refer to in the comment above only occurs within the "comm" command when run within Simon - the exact same script when run from Terminal (in a way that precisely simulates the sequence employed by Simon) does not manifest this bug. Let me know if you need further information.

Re: New Filter Variable: PreviousFilterPreviousText

Very much related to this issue is the "Difference" captured by Simon automatically, as seen in Simon's "Changes" pane.

However there doesn't seem to be a variable assigned to this difference - not a documented one that I can find anyway - nor does it ever seem to be the output of a filter that is set to be sensitive to change (not even the "Change" filter itself).

So my first question is: How can I capture this "Difference" displayed in Simon's "Changes" pane.

And my second question is: How is this "Difference" arrived at? Consider the following string:

^.*?[sub-string "A" that WASN'T present previously, but is now].*?[sub-string "X" that WAS present previously but isn't now].*?[sub-string "B" that WASN'T present previously, but is now].*?[sub-string "Y" that WAS present previously, but isn't now].*$

Your difference calculator might just return only the first addition (i.e. A), or it might return all additions (i.e. A B), or it might return all changes in full (i.e. +A -X +B -Y). So which is it? [EDIT: by experimentation it seems that it is A.*$ - this was perhaps a compromise due to complexity? Typically just A would be more useful - i.e. the difference calculator should detect where the old output resumes after the change, and a context filter can follow if required.]

It would be easy enough to program a logic that allows the user of Simon to choose a difference calculator as 99% of use-cases will be met with the following 8 possibilities. First give the option as to whether to treat the output as a single string or a set of sub-strings with a given separator between them (i.e. a list).

If a single string then give options for:
First addition only (i.e. A as above)
All additions (i.e. A B)
First subtraction only (i.e. X)
All subtractions (i.e. X Y)
All changes in full (i.e. +A -X +B -Y)

If a list then give options for:
Elements exclusive to current output
Elements exclusive to old output
Elements that are present in either just the current output OR just the old output (with a -/+ marker to differentiate them)

This can be scripted of course, as I have already done, but having that functionality built into Simon would add a myriad of applications that are currently not possible unless the user knows how to make their own such scripts.

Consider that a very high percentage of websites that people check frequently for updates are essentially just lists. Sure, some offer RSS feeds, but the majority do not and such a functionality would allow users to simply and easily be notified of new items on such lists that match a given criteria.

Yes, Simon does this already to some extent, but the use-cases are limited to watching for any change in a given element or filter of elements and outputting the whole lot for any given change (unless it knows in advance which items are going to be of interest so that the rest can be filtered out).

Simply watching the latest item(s) on a list is not the best solution either because not all lists have the latest additions in predictable places and, more importantly, the test has to be very frequent to mitigate the possibility of there being multiple additions to the list in a short space of time (resulting in not being alerted of any changes that fall beyond the scope of that filter).

My experience is that many of the things I want to be notified of fall outside of those use-cases, but that ALL of them would be satisfied with the addition of a difference calculator with the 8 options given above (which I've actually already accomplished via scripting, but I'm thinking of others)...

David Sinclair's picture

Re: New Filter Variable: PreviousFilterPreviousText

Quite a lot to go through there!

I agree that it would be very useful for Simon to have more sophisticated difference analysis options, perhaps even a source-text-style diff engine. At present, the difference analysis is very simple: it just compares the output of the last filter against the previous output, and finds the earliest and latest differences, and outputs the text between the two as the "Difference" (truncating to a reasonable length if necessary).

The output is available as a variable named {TestChangeText}. This difference text is available as {TestChangeDifferenceWithHTML} and {TestChangeDifferenceWithoutHTML}.

If you are able to create a filter script to do useful generic difference analysis based on the available variables, please share it so others can benefit.

If you want to help fund a feature enhancement to Simon to add better difference analysis, I offer paid enhancements as contract work.

Otherwise, I'll certainly keep this in mind as a possible area to improve for future versions.