30C3

We're getting together at the annual CCC conference, 30C3, for a sigrok hackathon. In addition to the usual "whatever we feel like hacking on", we also have several architectural decision to make, and doing this in person is a lot easier. If you want to be part of the conversation and decision-making process, show up!

In addition to the 4 days of the congress, we will also get together the day before (26 December 2013) for a hopefully less crowded and more productive day. Venue is not yet decided.

What's decided here is what goes.

New sigrok file format

New sigrok file format

Improved Configuration Enumeration

Improved Configuration Enumeration

Filtering/rearranging probes

To sr_filter_probes() or not, that is the question.

• frontends have probe location info, they pass it to sr_filter_probes() after all
• very nice for output modules: avoids having to map probe location to bit on all the modules
• waste of time for PV, which will soon do its own reformatting of logic data anyway
• SRD currently needs it, but has a probe mapping mechanism already, so can be adapted

• make sr_filter_probes() SR_PRIV
• sigrok-cli should not call it anymore
• every output module should call it for itself, since some may be fine without rearranged probes -- will never be called by (future) output module wrappers.

Driver scan

Rethinking how we enumerate and assign drivers to hardware

• Pre-scan for all resources such as serial ports & USB devices
• How users/frontends resolve ambiguity
• How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
• Device tree model?

Testing

• automated unit tests on commit (SR and SRD)
• VMs for *BSD, linux dists on kiutl for automated build/unit tests
• VMs with connected hardware (bert and uwe)?

Yes. Yes we can.

Malloc

Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [1], and even if we wanted to we could never do everything right because of the extensive use of glib [2]. So what is the use case we are actually targeting here?

Current glib usage is here.

We will:

• never check g_malloc returns for structs
• only check g_try_malloc returns for allocations which are/could be 1MB or over.

All g_malloc checks in the entire codebase (all projects) will be changed to conform to these rules.

• libsigrok
• libsigrokdecode
• sigrok-cli

This will not need to be done before the 0.3.0 release, but is a soft blocker for 0.4.0.

Progress

An API for allowing frontends to signal "operation in progress" better to the user.

• Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
• There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
  • Type 1: Operation in progress, but I do know when it will be finished. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
  • Type 2: Operation in progress, but I don't know when it will be finished. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
• Which operations can take a longer amount of time to complete?
  • Scan
    • Example: Various devices have to upload a firmware first before the scan can work, e.g. Saleae Logic and others. This upload can take a while.
  • Set/get/switch config options
    • Example: The Saleae Logic16 has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
  • Acquire data from a device
    • Example: The ChronoVu LA8 will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
  • Send data to a device
    • For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
  • Probably pretty much any other operation too, depending on driver and hardware.

Compression

RLE data as the internal currency for sigrok Logic.

• File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
• Many devices generate RLE. Those that don't are not extreme speed.
• RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
• libsigrokdecode needs to move to edge-transition processing for performance reasons.
  • Ergo... can we keep it RLE end to end?

Notes

1. Linux/glibc malloc() fails only when it runs out of address space; you'll get OOM killed long before malloc() fails.
2. Although glib offers g_try_malloc() which returns NULL on failure, all glib functions that allocate memory internally use g_malloc(), which calls abort() on failure. This is not considered a glib bug and will never be fixed. In many cases it couldn't even be fixed, without fundamental changes to the glib API, which lacks any mechanisms to signal malloc failure except for the simple g_try_malloc() case.

Release

Pre-release TODO:

• basic functionality check on every driver against real hardware (if available)
• public/private API:
  • go through headers
  • go through .so symbol table
  • doxygen
• NEWS file
• on release branch: downgrade libusb req to 1.0.9
• libserialport runtime version checking mechanism
• replace poll() usage in libserialport with select()

Jenkins

• libsigrok build (fix copy artifact plugin)
• doxygen build for sp/sr/srd