Difference between revisions of "30C3"

From sigrok
Jump to navigation Jump to search
Line 5: Line 5:
What's decided here is what goes.
What's decided here is what goes.


* [[New sigrok file format]]
== New sigrok file format ==
* [[Improved Configuration Enumeration]]
[[New sigrok file format]]
* To sr_filter_probes() or not, that is the question.
 
** frontends have probe location info, they pass it to sr_filter_probes() after all
== Improved Configuration Enumeration==
** very nice for output modules: avoids having to map probe location to bit on ''all'' the modules
[[Improved Configuration Enumeration]]
** waste of time for PV, which will soon do its own reformatting of logic data anyway
 
** SRD currently needs it, but has a probe mapping mechanism already, so can be adapted
== Filtering/rearranging probes ==
* Rethinking how we enumerate and assign drivers to hardware
To sr_filter_probes() or not, that is the question.
** Pre-scan for all resources such as serial ports & USB devices
* frontends have probe location info, they pass it to sr_filter_probes() after all
** How users/frontends resolve ambiguity
* very nice for output modules: avoids having to map probe location to bit on ''all'' the modules
** How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
* waste of time for PV, which will soon do its own reformatting of logic data anyway
** Device tree model?
* SRD currently needs it, but has a probe mapping mechanism already, so can be adapted
* Testing
 
** automated unit tests on commit (SR and SRD)
== Driver scan ==
** VMs for *BSD, linux dists on kiutl for automated build/unit tests
Rethinking how we enumerate and assign drivers to hardware
** VMs with connected hardware (bert and uwe)?
* Pre-scan for all resources such as serial ports & USB devices
* Malloc
* How users/frontends resolve ambiguity
** Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [0], and even if we wanted to we could never do everything right because of the extensive use of glib [1]. So what is the use case we are actually targeting here?
* How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
* API for allowing frontends to signal "operation in progress" better to the user.
* Device tree model?
** Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
 
** There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
== Testing ==
*** Type 1: '''Operation in progress, but I do know when it will be finished'''. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
* automated unit tests on commit (SR and SRD)
*** Type 2: '''Operation in progress, but I don't know when it will be finished'''. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
* VMs for *BSD, linux dists on kiutl for automated build/unit tests
** Which operations can take a longer amount of time to complete?
* VMs with connected hardware (bert and uwe)?
*** Scan
 
**** Example: Various devices have to upload a firmware first before the scan can work, e.g. [[Saleae Logic]] and others. This upload can take a while.
== Malloc ==
*** Set/get/switch config options
Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [0], and even if we wanted to we could never do everything right because of the extensive use of glib [1]. So what is the use case we are actually targeting here?
**** Example: The [[Saleae Logic16]] has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
 
*** Acquire data from a device
Current glib usage is [[Glib|here]].
**** Example: The [[ChronoVu LA8]] will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
 
*** Send data to a device
== Progress ==
**** For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
An API for allowing frontends to signal "operation in progress" better to the user.
*** Probably pretty much any other operation too, depending on driver and hardware.
* Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
* RLE data as the internal currency for sigrok Logic.
* There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
** File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
** Type 1: '''Operation in progress, but I do know when it will be finished'''. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
** Many devices generate RLE. Those that don't are not extreme speed.
** Type 2: '''Operation in progress, but I don't know when it will be finished'''. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
** RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
* Which operations can take a longer amount of time to complete?
** libsigrokdecode needs to move to edge-transition processing for performance reasons.
** Scan
*** Ergo... can we keep it RLE end to end?
*** Example: Various devices have to upload a firmware first before the scan can work, e.g. [[Saleae Logic]] and others. This upload can take a while.
** Set/get/switch config options
*** Example: The [[Saleae Logic16]] has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
** Acquire data from a device
*** Example: The [[ChronoVu LA8]] will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
** Send data to a device
*** For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
** Probably pretty much any other operation too, depending on driver and hardware.
 
== Compression ==
RLE data as the internal currency for sigrok Logic.
* File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
* Many devices generate RLE. Those that don't are not extreme speed.
* RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
* libsigrokdecode needs to move to edge-transition processing for performance reasons.
** Ergo... can we keep it RLE end to end?


====== Notes ======
====== Notes ======

Revision as of 16:07, 4 December 2013

We're getting together at the annual CCC conference, 30C3, for a sigrok hackathon. In addition to the usual "whatever we feel like hacking on", we also have several architectural decision to make, and doing this in person is a lot easier. If you want to be part of the conversation and decision-making process, show up!

In addition to the 4 days of the congress, we will also get together the day before (26 December 2013) for a hopefully less crowded and more productive day. Venue is not yet decided.

What's decided here is what goes.

New sigrok file format

New sigrok file format

Improved Configuration Enumeration

Improved Configuration Enumeration

Filtering/rearranging probes

To sr_filter_probes() or not, that is the question.

  • frontends have probe location info, they pass it to sr_filter_probes() after all
  • very nice for output modules: avoids having to map probe location to bit on all the modules
  • waste of time for PV, which will soon do its own reformatting of logic data anyway
  • SRD currently needs it, but has a probe mapping mechanism already, so can be adapted

Driver scan

Rethinking how we enumerate and assign drivers to hardware

  • Pre-scan for all resources such as serial ports & USB devices
  • How users/frontends resolve ambiguity
  • How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
  • Device tree model?

Testing

  • automated unit tests on commit (SR and SRD)
  • VMs for *BSD, linux dists on kiutl for automated build/unit tests
  • VMs with connected hardware (bert and uwe)?

Malloc

Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [0], and even if we wanted to we could never do everything right because of the extensive use of glib [1]. So what is the use case we are actually targeting here?

Current glib usage is here.

Progress

An API for allowing frontends to signal "operation in progress" better to the user.

  • Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
  • There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
    • Type 1: Operation in progress, but I do know when it will be finished. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
    • Type 2: Operation in progress, but I don't know when it will be finished. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
  • Which operations can take a longer amount of time to complete?
    • Scan
      • Example: Various devices have to upload a firmware first before the scan can work, e.g. Saleae Logic and others. This upload can take a while.
    • Set/get/switch config options
      • Example: The Saleae Logic16 has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
    • Acquire data from a device
      • Example: The ChronoVu LA8 will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
    • Send data to a device
      • For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
    • Probably pretty much any other operation too, depending on driver and hardware.

Compression

RLE data as the internal currency for sigrok Logic.

  • File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
  • Many devices generate RLE. Those that don't are not extreme speed.
  • RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
  • libsigrokdecode needs to move to edge-transition processing for performance reasons.
    • Ergo... can we keep it RLE end to end?
Notes
  1. Linux/glibc malloc() fails only when it runs out of address space; you'll get OOM killed long before malloc() fails.
  2. Although glib offers g_try_malloc() which returns NULL on failure, all glib functions that allocate memory internally use g_malloc(), which calls abort() on failure. This is not considered a glib bug and will never be fixed. In many cases it couldn't even be fixed, without fundamental changes to the glib API, which lacks any mechanisms to signal malloc failure except for the simple g_try_malloc() case.