29c3

We're getting together at the annual CCC conference, 29c3, for a sigrok hackathon. In addition to the usual "whatever we feel like hacking on", we also have several architectural decision to make, and doing this in person is a lot easier.

If you want to be part of the conversation and decision-making process, show up!

What's decided here is what goes.

Precision/accuracy analog format

We have a real problem with analog values at the moment. A new format is needed. Some options are listed on the High precision analog page.

Done.

Timestamps

Some devices, such as dataloggers, actually log the time at which a measurement was made. This is not the current time — some of these cannot even stream live data — so the driver needs to submit these measurements to the session bus with an attached timestamp.

Use cases

• Pure dataloggers -- devices which acquire measurements and log data in internal memory in a standalone fashion -- are sometimes able to log a timestamp as well. This may be per sample, or as a start time combined with a sample rate. An example of the latter is the Lascar EL-USB-* series. The timestamps are effectively in the past, and timestamps included in the data stream on the session bus are the only way to support this.
• Devices that log a timestamp with every measurement taken, e.g. data acquisition boards. The samplerate may or may not be fixed, but the timestamps are canonical.

Not applicable

• Logic analyzers with a fixed/configurable sample rate. Since the frontend knows when it started acquisition and when the first SR_DF_ANALOG packet came in, it can derive the timestamp of any sample by starttime + samplecount * samplerate.
• Oscilloscopes and spectrum analyzers send non-contiguous frames of data, where the elapsed time between frames can depend on many things; exact time of any given sample isn't really relevant.
• Slow-logging devices such as DMMs, environmental meters and so on have a vague and often unspecified samplerate, generally under 10Hz. The timing of these samples is often made worse by slow and unreliable serial connections with optical isolation. The frontend logging the time when a measurement comes in is thus the best that can be done for those devices.

We're missing some functionality in glib. The new GDateTime data type is interesting to work with, but it's an opaque type, so we cannot use it for transporting across the session bus inside an sr_datafeed packet. g_get_monotonic_time() returns a gint64, which is ideal for transporting (but should be unsigned). However there's no way to convert from GDateTime, or indeed from anything but the RTC, to that gint64. Converting from that gint64 to readable time can be done with glib, but would truncate to second precision.

Is there anything like a time handling lib that can do all this stuff properly, preferrably with uint64_t datatypes?

Chained packets

It may be interesting to link several packets together, and submit them to the session bus at the same time. For example, a timestamp + temperature measurement + relative humidity measurement submitted together would denote that these two measurements were taken at that specific point in time.

The two timestamp use cases above require this, so we'll do it.

TODO:

• come up with the structure for it (Bert)

Samplerate: meta or DI?

Samplerate is a part of the SR_DF_META_LOGIC packet. However there is also a SR_DI_CUR_SAMPLERATE info type. Do we really need two?

Use-cases for SR_DI_CUR_SAMPLERATE:

• Needed for sure, cannot be removed.
• Setting the samplerate might not always set it to the value the frontend wants.
  • Use case 1: "Set 1MHz", but HW actually sets to 0.9995MHz
  • Use case 2: "Set 14MHz" but sigrok driver knows hardware cannot do 14MHz and rounds to 16MHz. SR_DI_CUR_SAMPLERATE returns actually set samplerate. It needs a way to notify the frontend of this, though (see below).

meta_logic:

• GSList of key/value of what should actually change
• key ($thing to change), value: void*

Example:

samplerate, &uint64_t
num_probes, &uint64_t

Advantages:

• Common handling of any parameter changes the driver wants to notify the frontend about (samplerate, numprobes, ...)
• No API or ABI breakage when new parameters would be needed.

TODO:

• keys are enums (10000 apart)
• value is void* to enum-specific types/struct

Multiple ways/types to specify same thing (samplerate/period/divisor):

• method 1: multiple enums, type is implied for each enum
• method 2: sr_devopts-like list of specific parameters + description with either SR_T_CHAR etc. or (later) sr_opt based types. Then: meta_logic goes from (enum,void*) to (enum,type,void*)

Unrelated:

• meta_logic / meta_analog -> one meta (generic)

HWCAPS:

• SR_HWCAP_MULTIMETER etc. -> used by GUIs later to know what kind of device it is, so that it can show the data differently etc.
• sr_hwopt -> sr_config
• sr_hwcap_option -> sr_config_info
  • hwcap -> key (enum)
  • shortname -> id
  • desc -> name
  • add description with longer sentence
  • int type -> int datatype (SR_T_FLOAT etc.) -- will be sr_opt later (sr_opt datatype)

Done.

Supporting devices/formats with more than 64 probes

We're currently limited to logic analyzers (and other devices) with a max. number of 64 probes. However, there are also devices with a lot more than just 64 probes. Independently of such hardware, we also want to support input/output file formats which can have > 64 probes and protocol decoders with > 64 probes.

This issue needs to be addressed in libsigrok and libsigrokdecode and in the frontends.

• libsigrok defines SR_MAX_NUM_PROBES to 64. This limit shouldn't exist at all. Uses in libsigrok:
  • input/vcd
  • output/gnuplot
  • output/text
  • output/chronovu_la8
  • output/csv
  • output/vcd

These uses mostly come down to static arrays, which can be made dynamic.

• pulseview has an assert that uses it. Is this due to a uint64_t used to store sample values?
• sigrok-cli uses it in an array, which can be made dynamic.

Better support for controlling devices

The current API focuses on getting samples (logic/digital or analog) from devices such as logic analyzers, multimeters, oscilloscopes, and so on.

For the upcoming support of more "control-oriented" device types such as power-supplies, function generators, GPIB interfaces, and others we'll need additional, generic API functions that allow specifying various controllable parameters, their types, allowed values, and so on.

New sigrok file format

The current .sr file format is a ZIP file containing multiple files (some metadata files and actual sampling data files). This works fine, but it also has some issues:

• In order to get to the data you want, you need to decompress the whole file.
• Appending to a file is not possible easily (and it's not efficient).
• ...

Wish list for the new file format:

• It should be able to store metadata and arbitrary data (logic samples, and/or analog samples, and so on).
• It must support compression.
• It should be able to handle run-time changes in the data streams (via meta packets on the session bus), e.g. changing samplerates, changing probes, etc. etc.
• Better compression properties (e.g. using LZO or other algorithms, this is to be evaluated). What we ideally want out of the compression algorithm is:
  • Good and relatively fast compression results at only moderate CPU usage.
  • Very fast decompression (LZO is probably the best one here, as it's specifically designed for this).
  • Ideally, support for appending further data to already compressed data chunks (though this could be also implemented outside of the compression algorithm per se).
  • Open-source license and OS portability. There should be an open-source library or code chunk for compression/uncompression and it should be widely available in Linux distros, and portable to Windows, Mac OS X, FreeBSD, and so on.
• Independent of hardware architecture (x86, ARM, PowerPC, MIPS, and so on), OS, endianness, float representation, and so on. All data fields must be properly specified (endianness, signedness, size, format).

• Must be (optionally) possible to store extra UI state data. E.g User configured probe colours, names, positions.

Update: The specification page for the new file format.

libusbtmc

• Properly understand what's there now
• Sketch out the general shape of things to come and at least the API for basic functionality
• Stubs for all public API functions
• Define complete structs for all USBTMC packet types

libusbtmc is out, we're using VISA instead. See 29c3#GPIB/VISA

Scanning hints

• conn: HWOPT_CONN_[USB|SERIAL]_[OPTL|REQD] ?
• serialcomm OK?
• spec formats

We'll use SR_HWCAP_CONN_USB and SR_HWCAP_CONN_SERIAL, and use Uwe's option metadata work to specify things like "the frontend needs to specify either one or the other".

serialcomm is OK -- if it's there it's optional, driver should work without it as it knows what serial comm parameters its supported devices use.

TODO:

• Finish and document option metadata work (Uwe)
• Specify conn USB and serial format specifation on the wiki (Bert)

Domain-specific measurements and analysis

Some measurement gear (oscilloscope, specans) have specific measurements and analysis done to their streams by most/all devices. For example any oscilloscope with a display or GUI will support Vpp, rise time, Vmin/max, and so on. Any spectrum analyzer will have peak search, and a few spectrum filters.

If libsigrok supports such measurements as part of its stream or API, this would mean:

• not every frontend needs to write that functionality
• if some piece of hardware does this, libsigrok can just pass it on

We need to figure out how to support this.

---

• should be optional, enabled in the driver
  • there needs to be a way to enumerate a list of measurements the driver supports supplying, based on the type of device it supports
  • a way to enable them one by one
• the actual functionality should not be in the driver, but rather in a generic way in libsigrok. The session loop should intercept the DF packets and feed them to the various measurement modules, which put measurement/analysis packets on the session bus as needed. Therefore the list should be supplied by the driver in response to a simple info query like SR_DI_HWCAPS and SR_DI_NUM_PROBES. Enabling of these measurements shouldn't touch the driver at all, but rather be intercepted as a libsigrok setting
  • however some hardware may support some of these in hardware, in which case the driver DOES need to support it (ask Simon for specifics).
    • libsigrok will know how to interpret some of the enums (e.g. generic Vpp), but not others (e.g. "hardware-based Vpp"). Those it doesn't support will therefore be enabled in the driver, and the driver will send its own measurement packet to the session bus, based on what the hardware provided.
    • something like SR_AN_VPP (libsigrok) and SR_AN_HW_VPP (hardware-based)
• we thus need:
  • enum for measurement list (SR_DI_MEASUREMENTS or SR_DI_ANALYSIS?)
  • a packet type for analysis
  • some enumeration of analysis types (Vpp, risetime, falltime, ...) for use as a field in those packets and enum towards the frontend for the DI, similar to how SR_DF_ANALOG has mq/unit/mqflags fields.

---

This should be integrated with:

• Factor out libsigrok filter into modular transform system.
  • ADC transform, option TTL/CMOS/RS232: transform SR_DF_ANALOG to SR_DF_LOGIC.
  • take arguments in the regular thing:key=value format
  • integrate current probe compression filter
  • resampling module: sample up/down based on factor argument
  • noise filter: filter out pulses shorter than the samplerate's period
  • software triggering
    • integrate current basic trigger functionality from saleae driver
    • for streaming devices, opportunity for more complex triggers than the hardware can do
    • We will have an extra datafeed type SR_DF_LOGIC_INDEXED, which contains [samplenumber, sample] instead of just raw samples. Samplenumber is a uint64_t, and sample is unchanged (unitsize). Frontends must be able to handle both, and modular filters in libsigrok or the sampling filter in libsigrokdecode may convert from SR_DF_LOGIC to SR_DF_LOGIC_INDEXED at any time.

---

We will implement both of these as a new modular framework, with an API similar to the new output format API: setup, cleanup, recv. There should also be a way for the module to declare whether it's a filter, i.e. it gets a feed and submits it back possibly altered, or it wants to get a duplicate feed and do its own analysis on it, and possibly submit analysis stuff to the session bus. This is fundamental to what the module does, so can be a static declaration in the registration struct. TODO:

• name for the modular framework? filter/transform/measurement/analysis

GPIB/VISA

Two ways in which sigrok can be useful:

• We're writing GPIB drivers for a bunch of common USB/ethernet-connected GPIB interfaces. We want to make those drivers useful for more than just the sigrok project. A good way to do this is to make them available via a VISA library. We don't need to write one: Simon's freevisa (name?) library is in active development, and will natively support ethernet-connected SCPI devices, i.e. LXI. We can make a plugin for that library that registers as e.g. "GPIB0" and backends on libsigrok, translating VISA API calls into libsigrok calls.

• We will write a GPIB backend resource to librevisa instead, and make low-level GPIB interface drivers for that. libsigrok will have a librevisa backend, which will give us access to its GPIB and USBTMC backends via resource strings.
  TODO:
  • librevisa GPIB backend framework (Simon)
  • ICS 488-USB, GPIB-USB 82357B clone, NI GPIB-ENET, NI GPIB-USB-HS (Bert)
  • Beiming S82357 (Uwe)
  • Prologix GPIB-USB (Robert)

• Upper-level drivers that typically sit on top of a VISA library, know everything about a particular device, and are written by that device's vendor implement an IVI API. That API (e.g. IviScope) is specific to a certain class of device, but vendor-independent. Since libsigrok supports lots of devices for which no IVI API or VISA layers are available, we can write an IVI API library that actually backends on libsigrok, exposing all our supported devices to software that only needs to know one of the IVI APIs.