Input Collection Filter


Have you ever run into a situation where you have multiple different files that you want to do the same thing to, whether to plot them or to perform some analysis to calculate a value? This works well until you realize that you want to see all that values at once. This filter has been added to address this situation. The filter itself simply collects elements from the input and "remembers" values over multiple times of having the input change. To give a specific example, consider a set of large simulations where for a particular analysis you only care about a small set of the particles such as all those that are within a certain short distance of one particular body. This is particularly relevant for simulations that output a separate file for each time step as would often be the case in ring simulations. In that situation, the whole simulation might include millions of particles, but for this analysis you only care about a few thousand in each file. Unfortunately, you have to read in the file to figure out which ones you care about because you have to compare bodies to something else from the file.

The "standard" solution to this would be to include multiple sources and have them all go to filters that filter out the stuff you don't want. Unfortunately, just loading in all the data sets could cause memory problems. If the number of files is really large, there are other problems. The input collection filter provides a way to get around these problems.

You could have a single source, with a selection filter after it that removes everything but the elements you are interested in for this analysis. Have that go to an input collection filter and then read in each of the files that you are interested in. The input collection filter will keep the selected elements for all of the times the source is read and tag each with a parameter that grows each time the input changes. These could then be split out or dealt with in some other way. The input collection filter could also be put further down stream in the graph so that it only stored calculated values instead of all the particle of interest. The figure below shows the options for this filter. It is remarkably simple with only a single integer and a button to reset. The integer tells the filter how many changes to keep track of. Much of the time you will want this number big enough to hold all the inputs. However, you can set the number smaller to get a "boxcar" effect where the older elements are thrown out after a certain number of updates have occurred.

If the number of data files is larger than what you would want to select by hand there is a feature of the Auto Process option that can be used. If the image file name is set to be blank, all the files will be read, but no images will be output. This can be used as an easy way to go through a large number of files that can be collected in an input collection filter.

There is another feature that the input collection filter can be used with that will not be significant for most SwiftVis users, but can have serious implications for some uses. The socket source is a source that reads data off of a network socket. This allows simulations to send data directly to SwiftVis as they are running. The elements in this source are completely overwritten each time that a new set of data is sent across the network. The input collection filter can be used as a means to compile multiple sets of data that have come across the network.