34C3

We're getting together at the annual CCC conference, 34C3, for a sigrok hackathon. In addition to the usual "whatever we feel like hacking on", we also have several architectural decision to make, and doing this in person is a lot easier. If you want to be part of the conversation and decision-making process, show up!

New sigrok file format

File format:sigrok/v3

What's missing to make it happen?

Discuss concepts for sparse data loading presented here: https://sourceforge.net/p/sigrok/mailman/message/25153496/

[2017-12-28]

Martin, Uwe and Soeren discussed the current state of the spec. In general it's fine and can be implemented, some remarks:

• All packets will have a 1-byte version field to allow changing the spec on a per-packet level without the need for new UUIDs (Rationale: an older client will silently ignore packets with unknown UUIDs but a known-UUID packet with an unrecognized version allows the client to produce a meaningful error message to the user)
• A field that is mandatory is added to the relevant packet, a field that's optional receives its own packet (Rationale: when changing the meaning of a field, increasing the version of the packet it belongs to would result in an unreadable file for older clients if the packet is mandatory. If a client encounters an optional packet with an unknown version, it can simply be skipped and the file remains readable)
• PV will use custom UUIDs to store UI/acquisition/device state with saved session files
• PV will use the same file format to store UI/acquisition/device state, just without any device/channel/data packets

Handling of RLE data

What is needed to make libsigrok able to handle RLE data? Should there be a common format that drivers have to convert custom formats to? Should there be a new session bus packet type? What are the implications?

See also the discussion from 30C3 here.

[2017-12-30]

Postponed until after the flowgraph API is stable and we have feature parity with the current state of affairs.

Analog I/O for libsigrokdecode

What are the requirements/use cases? What's needed to make it happen?

[2017-12-29]

Uwe and Soeren discussed this. Results:

• PV will register two additional callbacks with SRD: one for "new analog channel" and one for "new analog data"
• When a PD creates an analog output, SRD will register this analog output using the callback and provide the metadata (e.g. unit)
• When a PD produces analog output data, SRD will call the "new analog data" output and submit a list of key/value pairs: key being the absolute number of the input sample where the analog value came into existance and value being the analog value at this point in time
• Analog input will be discussed separately, we'll do analog output first
• PDs will also need to be able to emit logic traces in the future; the process will be similar to analog
• Declaration for ann classes with data types may look like this: ('temperature', 'Temperature in degrees Celsius', ('datatype':FLOAT, 'unit':'C'))
• This change will require an libSRD API change, so this will be libSRD API v4

Flowgraph architecture

How do we achieve this step-by-step? It won't ever happen if it's a huge change :) Currently, plans are to make PV have a modularized internal structure and use flowgraphs internally to gather experience and have something working

[2017-12-27/28]

This is what we have:

• Martin wrote a python PoC for some flowgraph architecture ideas
• Soeren found usable flowgraph UI code: https://github.com/paceholder/nodeeditor

This is what we want:

• libsigrokflow as a core library that provides the basic flowgraph architecture elements (potentially re-usable by different projects)
• The same library allows libsigrok and libsigrokdecode to be tied into the flowgraph architecture
• Library code in C/C++ for speed and flexibility to allow blocks to be written in any language later on
• numpy BitArray-like data type for uncompressed data, allowing blocks to hand over BitArray objects if they don't change it ("it's a view on the raw data")
• One thread per source block, downstream blocks are called within the same thread
• FIFO blocks that act like a buffer+source block, creating a new thread to allow decoupling
• Signal (connection between two blocks) types are modeled by hierarchical C++ classes, e.g. Analog;Logic->Uncompressed/RLE;Decode to allow type checking at runtime and easy "don't care" block inputs: if a block accepts Logic, can just tell the Logic base class to hand over RLE data without needing to worry about any conversions required to obtain it
• libsigrokgui as a library that contains Qt-based UI elements used to build a basic sigrok client UI; at least a sigrok flowgraph UI window (including all necessary libsigrokflow interfacing) and trace drawing widgets

This is how we'll do it:

• We'll work on top and bottom levels at the same time
• First, Martin and Soeren will work on a python-based PoC, including a throwaway UI to test out the API
• When the architecture is stable, Martin will work on the C++ framework for libsigrokflow and Soeren on libsigrokgui and its PV integration
• Uwe will set up the subproject on git once the library is in a usable state (no longer PoC status)

Extra thoughts:

• Blocks have key/value properties, keys being strings allowing the object to provide descriptions as well
• This way, we can get the https://sigrok.org/wiki/Improved_configuration_enumeration for free
• libsigrokgui is needed because otherwise, we will never see full-featured UIs - the UI code would be too tightly coupled to PV; if not doing this, PV will be regarded as the universal sigrok client and see more and more feature creep

This needs more thinking:

• How would a plugin mechanism work for the blocks?
• How can blocks be made available at runtime? (i.e. device blocks)
• How do we handle device discovery?
• Do new signal types need to be addable at runtime, e.g. for PD blocks that provide binary data (e.g. images or EEPROM dumps) or do we simply use a "binary" type with metadata as properties, e.g. a MIME type to identify the contents?
• The current concept implies a push architecture where data is pushed out by the source and all attached blocks are only executed once they have enough data to operate on. However, for even moderately fast USB3 acquisition devices (think 400 MS/s per analog channel), this model breaks down. In this scenario, it would make more sense to let the acq driver buffer the data in the device's native format for performance reasons (data processing would break acquisition) and then let the consumers of the data (view, PD, etc.) pull the desired data from the connected blocks which in turn pull the data from the source, which then converts the device's native data to standardized data as it's requested. We need to decide which approach works better in the long run and determine how the decision affects our design.

Device threads

[2017-12-30]

Discussion resulted in the understanding that with the flowgraph API, we'll get device threads for free since every source block will run in its own thread and thus require its own libsigrok context. Currently, this doesn't work because libsigrok can't handle more than one context. This is a bug which needs fixing.

However, fixing this right now would allow clients to use one thread per device already, making device threads possible even before the flowgraph API stuff.

The one thing that needs some consideration is the device scan functionality. However, we wanted to rework that anyway, so there's now even more incentive to give this a higher priority.

PV: Definition of scope

What is PV supposed to be? What is it not supposed to be?

We should define this so we can determine a sane architecture and prevent feature creep. The architecture for PV would need major extension if we also want to support power supplies and frequency generators to be able to do (cool) stuff like network analysis, bode plots, load tests, pattern/signal generation and similar. However, the question is: what is realistic and what is better left to more specialized clients? Turning PV into a one-size-fits-all client will be too much work for one person and progress will slow down.

[2017-12-29]

For now, PV will only cover devices that deliver sampled data in regular intervals. Other devices (e.g. AWGs, pattern gens, PSUs) will be supported through libsigrokflow by means of input/output blocks.

PV: Scripting

Is it useful to have scripting available in PV? What use cases are there? Determine what would need to happen to support it.

[2017-12-29]

As there are more pressing issues in the near future, this isn't going to be on the roadmap for a while. However, it is worthwhile to check how scopy did it and implement this for PV as well since it appears to be pretty much free if Qt properties are attached to objects.

PV: Decode table view

The decode table view isn't going to be very usable at the moment because it lacks metadata that libsigrokdecode should provide. We should discuss what the decode table view should offer in terms of functionality and what's needed in libsigrokdecode to make it happen.

[2017-12-29]

Uwe and Soeren discussed use cases and the resulting requirements on the libsigrokdecode side:

• Simple key/value (key=annotation class, e.g. CRC) pairs are useful but insufficient
• PDs that output analog data must provide metadata for creation of libsigrok analog packets (e.g. unit)
• PDs should be able to define a "desired output format" for each annotation class - e.g. ASCII for integer annotations
• PDs could provide hierarchy information by referencing annotations by ID; this is a good way to deal with annotations which are generated as part of a packet but have a more global meaning and shouldn't be grouped with the packet itself
• PV should allow to jump to the prev/next selected annotation class
• PV should allow to configure all annotation classes: on/off, display format type (dec/hex/ASCII), etc.
• For now, we'll proceed without hierarchies

PV: Scope UI

PV is getting closer and closer to being usable for analog scopes. We should discuss ideas how a UI could look like that allows for full remote-control operation. There are good and bad UI examples to draw ideas from.

PV: Create "wishlist" wiki page

Users don't know what features are planned and when. Soeren has his own (non-public) to-do list and there are some items in the bugzilla. This should be unified so that users can see what's planned and what's not on our radar yet.

[2017-12-30]

Soeren will do the following:

• Remove "Roadmap" from the front page
• Move the items on "Roadmap" to "Todo" and turn "Roadmap" into a redirect to "Todo"
• Clean up the "Todo" page and link to the other locations where todo items are listed (e.g. decoder page, driver page and such)
• Tag all "wishlist" items in bugzilla with a proper flag
• List to a search query in bugzilla that lists all wishlist items

The sigrok workbench

[2017-12-30]

Further discussion of the whole sigrok stack brought up some points that showed us that in the end, we'll have the sigrok workbench.

• PV's UI is created on-the-fly using hard-coded C++ Qt API calls, making the UI inflexible and rigid
• Thus, PV UI elements are too tightly coupled to PV internals to be reusable easily and breaking them out into libsigrokgui makes little sense
• Instead, the widgets that go into libsigrokgui will be QML-based for great reusability, flexibility and complete Qt integration (also, we immediately get OpenGL support for free)
• This will allow the creation of a universal GUI client called the sigrok workbench
• UIs will be dynamically creatable and configurable and saved/loaded from UI preset files
• PV will be just a UI preset
• This approach allows specifying UI presets on the command line, virtually turning the workbench into a custom sigrok client
• To make this possible, UI elements need to be able to connect to libsigrokflow blocks via UI ports
• This means that UI ports may be difficult to implement if we want to support UI libraries other than Qt (e.g. Gtk+, Wx) but it's the only way forward; if there is no other way, we will limit the UI toolset to Qt
• In the long run, sigrok will support the current API for existing clients while the flow API will slowly become the standard
• For PV this means that it will be supported but once we have reached feature parity with the workbench, it will become a legacy client and will no longer see further development

As a first step, Soeren will check out QML in detail and use the libsigrokflow prototype UI as a test bed. Afterwards, he'll start implementing QML UI widgets in libsigrokgui that may or may not be integrated into PV.