High precision analog

Problem statement

We have a problem with the current SR_DF_ANALOG packets: the analog numbers are passed along as floats -- platform-specific versions of the C float type. The following summarizes what we need to solve:

Some numbers are just plain wrong

A floating point type is well known to be unable to store accurate representations of some numbers, as mentioned here. This hits us hard. In one driver the analog value is extracted from a device packet by dividing the derived value by 10, to convert to the single digit after the decimal point that is always implied. A value of "354" turns into not 35.4, but 35.400002; 353 turns into 35.299999. Note that these numbers are representations of the floats with printf()'s default precision of 6 digits after the decimal point; using ".20f" produces 35.39999999999999857891.

The numbers need to be stored and transmitted exactly as they were input by the driver.

Resolution

Understanding precision

Precision is casually defined as how close together a series of measurements of the same quantity are. However, this definition is confusing when one speaks of the precision of one measurement. A meter stick with markings 1 cm apart can only produce a precision of 1 cm. For example, a measurement of 23 cm implies that the value is anywhere between 22.5 cm and 23.5 cm. On the other hand, a stick with marking 1 mm apart will produce a more precise measurement: 23.0 cm implies that the value is anywhere between 22.95 cm and 23.05 cm.

It might be tempting to think of these measurements as:

1 cm precision: 23 cm ± 0.5  cm (WRONG!!!!)
1 mm precision: 23 cm ± 0.05 cm (WRONG!!!!)

The correct way to denote precision is via the number of significant figures presented with the measurement:

1 cm precision: 23 cm   (CORRECT - implies measurement is between 22.5 cm  and 23.5 cm )
1 mm precision: 23.0 cm (CORRECT - implies measurement is between 22.95 cm and 23.05 cm)

It is extremely important to note that trailing zeroes are significant. The precision is inferred from the number of significant digits.

The sigrok dilemma

Returning to our problem, a floating point data type has no notion whatsoever about the precision of the measurement. It has an intrinsic precision related to the size (number of bits) of the mantissa, which has nothing to do with the precision of the measurement.

Consider a device which always outputs a single digit after the decimal point; there is no more information beyond that digit. But we have no way of communicating that fact to the frontend. The float type doesn't store this information along with the value, and indeed changes the input value in such a way that the value can't even be truncated so as to make the resolution obvious (or irrelevant). There is simply no way that a frontend can receive 35.400002 and conclude that 35.4 is the proper number to display. Removing trailing zeroes is not a solution, as a value of 35.0 might incorrectly be truncated to 35.

While a value such as "35.4000" might reasonably be displayed as "35.4", it is nevertheless helpful (and common, in test & measurement gear) to display numbers at their full resolution; this gives a visual indication of that resolution. A frontend shouldn't have to assume or truncate anything; it should have exact information about what the device provided.

We need precision information transmitted along with the value.

Accuracy

Do not confuse accuracy with precision.

Understanding accuracy

Accuracy refers to how close a measurement is to the value being measured. A very common (and correct) way to describe accuracy is the uncertainty of a measurement

Offset error

let's return to our meter stick example. The meter stick with 1 mm markings might be missing a 2 cm chunk. It cannot be accurate; the measurement in that case would be presented as:

23.0 cm ± 2.0 cm

Note that although the precision is 0.05 cm, the accuracy is completely different.

Linearity error

A meter stick might, for example be calibrated at 25 C, but specified to be accurate from -10 C to 60 C. Due to contraction and dilation, at -10 C the 23.0 mark might be closer to 22.8 cm, while at 60 C, it might be closer to 23.3 cm. In this example, temperature is an external factor which affects the final measurement. It is unreasonable or impossible to estimate the error exactly, especially since the error grows linearly with the size of the measurement. This error is usually included in the specifications of the device, as an unknown. In this case it would be reasonable for the accuracy to be specified as

1.5 % of measurement

So that

at 23.0 cm, the value is 23.0 ± 0.3 cm
at 86.0 cm, the value is 86.0 ± 1.2 cm

Again, the accuracy and precision are not the same.

The sigrok dilemma

The value supplied by a device may differ from its resolution -- it may supply 35.4, but have a resolution that makes this 35.400 -- but this can still differ from how accurately the device can make a measurement. This accuracy is rarely supplied by the device along with the measurement, but commonly stated in the device's specifications. For example, The Fluke 187 has a DC voltage measurement specification stated as

Accuracy	 ± (0.025%+5)
Max. Resolution	 1 µV

This defines the accuracy of the measurement as 0.025% of the value plus 5 "counts" (least significant digits on the display). The true value can be anywhere up or down by this amount from the displayed value.

The given maximum resolution is not the same for every range. The Fluke 187 has a resolution of 50000 counts, so in order to get the maximum resolution of 1 µV, the device must be set to the 50 mV (50000 µV) range. Since the device can only distinguish 50000 values, a measurement of 1.3259, for example, will have a resolution of 0.1 mV; the device will not be able to distinguish voltage differences below that value. If it could, then the measurement would be 1.325900, and the device would have to be a 5000000 (5 million) count.

Expanding on our previous example, a measurement on 1.3259 has an accuracy of ±0.0336475 — making it somewhere between 1.2922525 and 1.3595475. The resolution, however, is 0.0001 (0.1 mV), so the last significant digits are meaningless. Note the displayed value is still limited to four digits after the decimal point! The correct way to communicate this measurement is:

1.3259 V ± 0.0337 V

The obtained value for the accuracy is almost always rounded up.

Hardware accuracy is known to the driver through knowledge of which device it's communicating with, and should be transmitted to the frontend -- including "extra counts" accuracy specifications.

Displayed value

The value displayed on the device may be limited by the number of digits available on a 7-segment display, and may well have less resolution than the device can handle. What libsigrok receives from the device may thus be a number with extra digits tacked on, compared to what's on the display. This may or may not be beyond the resolution or accuracy of the device. Nevertheless libsigrok receives the number and will transmit it to the frontend. It may be useful to a frontend to know how many digits are actually displayed on the device, however, if it chooses to mimic that display in some way.

Should the displayed value resolution be transmitted to the frontend?

The displayed value should always be transmitted to the frontend unaltered, unless the device datasheet or manual specifies that the precision of the measurement is higher than the one displayed, and the full precision is communicated through the PC interface.

Proposed solutions

Exact decimal representation

struct sr_decimal {
	int64_t value;
	int32_t pow_ten;
};

The sr_decimal is an exact representation of a measurement. The significand, or mantissa is kept in base 10, making it immune to all issues that a base 2 representation has. Its value is value * pow(10, pow_ten).

Advantages:

• Exact representation of measurement
• Immune to floating-point inconsistencies and rounding issues

Disadvantages:

• Very difficult to work with
• Not really usable as a math type (usually requires conversion to double beforehand)
• Requires a large amount of memory -- usually more than twice of the equivalent ASCII representation.
• Requires complex printf logic
• Signaling conditions such as NAN or INF requires helper functions
• Waste of storage space -- most of the bytes will usually be 0

Inexact floating-point representation with precision metadata

struct sr_measurement {
	double value;
	uint8_t sigfig;
};

Resoures

• Wikipedia: Accuracy and precision
• Fluke: Understanding specifications for precision multimeters (PDF)
• National Instruments: Digital Multimeter Measurement Fundamentals
• Keithley: Low-level measurements handbook (PDF)
• Agilent: How to select a Handheld DMM that is RIGHT for you (short info on "resolution, digit, and accuracy") (PDF)