From: Gerhard Sittig <redacted>
Date: Thu, 3 Oct 2019 16:50:21 +0000 (+0200)
Subject: scpi-dmm: prefer double over float data types more often
X-Git-Url: https://sigrok.org/gitaction?a=commitdiff_plain;h=a5346baf78a04f0a749dca9669ebc1f4331db3f7;p=libsigrok.git

scpi-dmm: prefer double over float data types more often

The 'float' data type uses 23 bits for the mantissa, which can express
some 6.9 decimal places. Some meters claim "6 digits" when in fact they
are 6.5 or so. Averaging over higher numbers of power line cycles (NPLC)
could additionally raise a meter's precision. Prefer double over float
for 6-digit devices already, just to err on the safe side.

The previous implementation only started using the double data type for
devices with 7 and more digits, and left a theoretical gap that's fully
closed now.
---

diff --git a/src/hardware/scpi-dmm/protocol.c b/src/hardware/scpi-dmm/protocol.c
index f413fcd7..a13db841 100644
--- a/src/hardware/scpi-dmm/protocol.c
+++ b/src/hardware/scpi-dmm/protocol.c
@@ -396,7 +396,7 @@ SR_PRIV int scpi_dmm_get_meas_agilent(const struct sr_dev_inst *sdi, size_t ch)
 	if (ret != SR_OK)
 		return ret;
 	g_strstrip(response);
-	use_double = devc->model->digits > 6;
+	use_double = devc->model->digits >= 6;
 	ret = sr_atod_ascii(response, &info->d_value);
 	if (ret != SR_OK) {
 		g_free(response);