Difference between revisions of "30C3"

From sigrok
Jump to navigation Jump to search
 
(23 intermediate revisions by 3 users not shown)
Line 5: Line 5:
What's decided here is what goes.
What's decided here is what goes.


* [[New sigrok file format]]
== New sigrok file format ==
* [[Improved Configuration Enumeration]]
[[File format:sigrok/v3]]
* To sr_filter_probes() or not, that is the question.
** frontends have probe location info, they pass it to sr_filter_probes() after all
** very nice for output modules: avoids having to map probe location to bit on ''all'' the modules
** waste of time for PV, which will soon do its own reformatting of logic data anyway
** SRD currently needs it, but has a probe mapping mechanism already, so can be adapted
* Rethinking how we enumerate and assign drivers to hardware
** Pre-scan for all resources such as serial ports & USB devices
** How users/frontends resolve ambiguity
** How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
** Device tree model?
* Testing
** automated unit tests on commit (SR and SRD)
** VMs for *BSD, linux dists on kiutl for automated build/unit tests
** VMs with connected hardware (bert and uwe)?
* Malloc
** Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [0], and even if we wanted to we could never do everything right because of the extensive use of glib [1]. So what is the use case we are actually targeting here?
* API for allowing frontends to signal "operation in progress" better to the user.
** Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
** There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
*** Type 1: '''Operation in progress, but I do know when it will be finished'''. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
*** Type 2: '''Operation in progress, but I don't know when it will be finished'''. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
** Which operations can take a longer amount of time to complete?
*** Scan
**** Example: Various devices have to upload a firmware first before the scan can work, e.g. [[Saleae Logic]] and others. This upload can take a while.
*** Set/get/switch config options
**** Example: The [[Saleae Logic16]] has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
*** Acquire data from a device
**** Example: The [[ChronoVu LA8]] will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
*** Send data to a device
**** For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
*** Probably pretty much any other operation too, depending on driver and hardware.
* RLE data as the internal currency for sigrok Logic.
** File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
** Many devices generate RLE. Those that don't are not extreme speed.
** RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
** libsigrokdecode needs to move to edge-transition processing for performance reasons.
*** Ergo... can we keep it RLE end to end?


====== Notes ======
<font color="blue">
* Streamability
* Add arbitrary data (GUID, length, data)
* Store annotations
* Index at the end
* Use generic GUIDs where possible (colors, probe names, etc)
</font>


# Linux/glibc malloc() fails only when it runs out of address space; you'll get OOM killed long before malloc() fails.
== Improved Configuration Enumeration==
# Although glib offers g_try_malloc() which returns NULL on failure, all glib functions that allocate memory internally use g_malloc(), which calls abort() on failure. This is not considered a glib bug and will never be fixed. In many cases it couldn't even be fixed, without fundamental changes to the glib API, which lacks any mechanisms to signal malloc failure except for the simple g_try_malloc() case.
[[Improved Configuration Enumeration]]
 
== Filtering/rearranging probes ==
<s>To sr_filter_probes() or not, that is the question.
* frontends have probe location info, they pass it to sr_filter_probes() after all
* very nice for output modules: avoids having to map probe location to bit on ''all'' the modules
* waste of time for PV, which will soon do its own reformatting of logic data anyway
* SRD currently needs it, but has a probe mapping mechanism already, so can be adapted
</s>
 
<font color="blue">
<s>
* make sr_filter_probes() SR_PRIV
* sigrok-cli should not call it anymore
* every output module should call it for itself, since some may be fine without rearranged probes -- will never be called by (future) output module wrappers.
</s>
Done!
</font>
 
== Driver scan ==
<s>
Rethinking how we enumerate and assign drivers to hardware
* Pre-scan for all resources such as serial ports & USB devices
* How users/frontends resolve ambiguity
* How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
* Device tree model?
</s>
 
<font color="blue">
We will define a device tree model.
 
For scan operations, a worker thread will be started for each driver.
 
Drivers will create device instances and submit them to the device tree (protected by a single mutex).
 
Thus, scanning will be parallelized.
 
A pre-scan will perform common enumeration operations to avoid repeated/conflicting work in each driver.
</font>
 
== Testing ==
<s>
* automated unit tests on commit (SR and SRD)
* VMs for *BSD, linux dists on kiutl for automated build/unit tests
* VMs with connected hardware (bert and uwe)?
</s>
<font color="blue">
Yes. Yes we can.
</font>
 
== Malloc ==
<s>Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [1], and even if we wanted to we could never do everything right because of the extensive use of glib [2]. So what is the use case we are actually targeting here?
 
Current glib usage is [[Glib|here]].
 
<font color="blue">
We will:
* <b>use g_malloc for structs, no check required.</b>
* <b>only use g_try_malloc for allocations which are/could be 1MB or over.</b>
All g_try_malloc checks in the entire codebase (all projects) will be changed to conform to these rules.
* libsigrok
* libsigrokdecode
* sigrok-cli
This will not need to be done before the 0.3.0 release, but is a soft blocker for 0.4.0.
</font></s>
 
Done.
 
== Progress ==
<s>
An API for allowing frontends to signal "operation in progress" better to the user.
* Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
* There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
** Type 1: '''Operation in progress, but I do know when it will be finished'''. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
** Type 2: '''Operation in progress, but I don't know when it will be finished'''. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
* Which operations can take a longer amount of time to complete?
** Scan
*** Example: Various devices have to upload a firmware first before the scan can work, e.g. [[Saleae Logic]] and others. This upload can take a while.
** Set/get/switch config options
*** Example: The [[Saleae Logic16]] has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
** Acquire data from a device
*** Example: The [[ChronoVu LA8]] will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
** Send data to a device
*** For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
** Probably pretty much any other operation too, depending on driver and hardware.
</s>
 
<font color="blue">
 
We are going to address this by changing the thread model of libsigrok.
 
Each sdi will have its own worker thread.
 
Frontend-initiated operations will be queued for the worker thread to execute.
 
The frontend gets a handle when it submits each operation. The handle can be used to check or wait for an operation's completion.
 
At any time, the frontend can request what an sdi is doing, it will get back a data structure with a text description and where applicable a fraction representing progress so far.
 
</font>
 
== Compression ==
<s>
RLE data as the internal currency for sigrok Logic.
* File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
* Many devices generate RLE. Those that don't are not extreme speed.
* RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
* libsigrokdecode needs to move to edge-transition processing for performance reasons.
** Ergo... can we keep it RLE end to end?
</s>
 
<font color="blue">
We will allow different encodings of logic data in the feed.
 
We will define a good compromise RLE implementation as one of the options.
 
Others may be added later.
 
Clients will make queries for data via libsigrok, which will implement them as necessary.
</font>
 
== Release ==
<font color="blue">
Pre-release TODO:
* basic functionality check on every driver against real hardware (if available)
* public/private API:
** go through headers
** go through .so symbol table
** doxygen
* NEWS file
* on release branch: downgrade libusb req to 1.0.9
* libserialport runtime version checking mechanism
</font>
 
== Jenkins ==
<s>
* libsigrok build (fix copy artifact plugin)
* doxygen build for sp/sr/srd
</s>
 
<font color="blue">Yes.</font>

Latest revision as of 17:23, 1 October 2017

We're getting together at the annual CCC conference, 30C3, for a sigrok hackathon. In addition to the usual "whatever we feel like hacking on", we also have several architectural decision to make, and doing this in person is a lot easier. If you want to be part of the conversation and decision-making process, show up!

In addition to the 4 days of the congress, we will also get together the day before (26 December 2013) for a hopefully less crowded and more productive day. Venue is not yet decided.

What's decided here is what goes.

New sigrok file format

File format:sigrok/v3

  • Streamability
  • Add arbitrary data (GUID, length, data)
  • Store annotations
  • Index at the end
  • Use generic GUIDs where possible (colors, probe names, etc)

Improved Configuration Enumeration

Improved Configuration Enumeration

Filtering/rearranging probes

To sr_filter_probes() or not, that is the question.

  • frontends have probe location info, they pass it to sr_filter_probes() after all
  • very nice for output modules: avoids having to map probe location to bit on all the modules
  • waste of time for PV, which will soon do its own reformatting of logic data anyway
  • SRD currently needs it, but has a probe mapping mechanism already, so can be adapted

  • make sr_filter_probes() SR_PRIV
  • sigrok-cli should not call it anymore
  • every output module should call it for itself, since some may be fine without rearranged probes -- will never be called by (future) output module wrappers.

Done!

Driver scan

Rethinking how we enumerate and assign drivers to hardware

  • Pre-scan for all resources such as serial ports & USB devices
  • How users/frontends resolve ambiguity
  • How to handle "gateway" devices such as GPIB adapters, networked servers with instruments, etc.
  • Device tree model?

We will define a device tree model.

For scan operations, a worker thread will be started for each driver.

Drivers will create device instances and submit them to the device tree (protected by a single mutex).

Thus, scanning will be parallelized.

A pre-scan will perform common enumeration operations to avoid repeated/conflicting work in each driver.

Testing

  • automated unit tests on commit (SR and SRD)
  • VMs for *BSD, linux dists on kiutl for automated build/unit tests
  • VMs with connected hardware (bert and uwe)?

Yes. Yes we can.

Malloc

Decide whether to continue trying to handle malloc failures in libsigrok. These code paths are never tested, would never be executed anyway due to the realities of virtual memory systems [1], and even if we wanted to we could never do everything right because of the extensive use of glib [2]. So what is the use case we are actually targeting here?

Current glib usage is here.

We will:

  • use g_malloc for structs, no check required.
  • only use g_try_malloc for allocations which are/could be 1MB or over.

All g_try_malloc checks in the entire codebase (all projects) will be changed to conform to these rules.

  • libsigrok
  • libsigrokdecode
  • sigrok-cli

This will not need to be done before the 0.3.0 release, but is a soft blocker for 0.4.0.

Done.

Progress

An API for allowing frontends to signal "operation in progress" better to the user.

  • Various operations in libsigrok drivers can take a long time. It would be good if frontends could meaningfully handle those (instead of just blocking the UI).
  • There are multiple types of "in progress" states that can happen and that libsigrok should be able to report to frontends:
    • Type 1: Operation in progress, but I do know when it will be finished. There are various ways a frontend can signal this condition to the user (e.g. show a progressbar from 0%-100%), the exact method to show this is up to the frontend.
    • Type 2: Operation in progress, but I don't know when it will be finished. There are various ways for a frontend to signal this condition to the user too (e.g. show a spinning wheel, which only signals "in progress" but no information when the operation will be finished).
  • Which operations can take a longer amount of time to complete?
    • Scan
      • Example: Various devices have to upload a firmware first before the scan can work, e.g. Saleae Logic and others. This upload can take a while.
    • Set/get/switch config options
      • Example: The Saleae Logic16 has to upload a new firmware and/or bitstream when certain config options (number of channels, voltage thresholds, ...) are changed. This is not only at scan time, it can happen at any point when the user changes the options in the frontend.
    • Acquire data from a device
      • Example: The ChronoVu LA8 will always fill its internal 8MByte buffer with LA samples, and then transfer them over USB. This operation typically takes more than 30 seconds time. This is a "type 1" case, since libsigrok / the driver knows how many samples still have to be transferred until done.
    • Send data to a device
      • For some write/send operations (e.g. uploading a waveform to a function generator) can also take a lot of time, depending on the device.
    • Probably pretty much any other operation too, depending on driver and hardware.

We are going to address this by changing the thread model of libsigrok.

Each sdi will have its own worker thread.

Frontend-initiated operations will be queued for the worker thread to execute.

The frontend gets a handle when it submits each operation. The handle can be used to check or wait for an operation's completion.

At any time, the frontend can request what an sdi is doing, it will get back a data structure with a text description and where applicable a fraction representing progress so far.

Compression

RLE data as the internal currency for sigrok Logic.

  • File Storage: Logic can be most efficiently stored in a planar format, with RLE+Huffman/Golomb Compression.
  • Many devices generate RLE. Those that don't are not extreme speed.
  • RLE is an efficient format for GUIs rendering logic. Much faster than traversing 100s of uncompressed values.
  • libsigrokdecode needs to move to edge-transition processing for performance reasons.
    • Ergo... can we keep it RLE end to end?

We will allow different encodings of logic data in the feed.

We will define a good compromise RLE implementation as one of the options.

Others may be added later.

Clients will make queries for data via libsigrok, which will implement them as necessary.

Release

Pre-release TODO:

  • basic functionality check on every driver against real hardware (if available)
  • public/private API:
    • go through headers
    • go through .so symbol table
    • doxygen
  • NEWS file
  • on release branch: downgrade libusb req to 1.0.9
  • libserialport runtime version checking mechanism

Jenkins

  • libsigrok build (fix copy artifact plugin)
  • doxygen build for sp/sr/srd

Yes.

Retrieved from "https://sigrok.org/w/index.php?title=30C3&oldid=12781"