[PATCH v2] iio: dac: ad5592r: fix temperature channel scaling value

[PATCH v2] iio: dac: ad5592r: fix temperature channel scaling value

[marc.ferland]

From: Marc Ferland <marc.ferland@sonatest.com>

The scale value for the temperature channel is (assuming Vref=2.5 and
the datasheet):

    376.7897513

When calculating both val and val2 for the temperature scale we
use (3767897513/25) and multiply it by Vref (here I assume 2500mV) to
obtain:

  2500 * (3767897513/25) ==> 376789751300

Finally we divide with remainder by 10^9 to get:

    val = 376
    val2 = 789751300

However, we return IIO_VAL_INT_PLUS_MICRO (should have been NANO) as
the scale type. So when converting the raw temperature value to the
'processed' temperature value we will get (assuming raw=810,
offset=-753):

    processed = (raw + offset) * scale_val
              = (810 + -753) * 376
	      = 21432

    processed += div((raw + offset) * scale_val2, 10^6)
              += div((810 + -753) * 789751300, 10^6)
	      += 45015
    ==> 66447
    ==> 66.4 Celcius

instead of the expected 21.5 Celsius.

Fix this issue by changing IIO_VAL_INT_PLUS_MICRO to
IIO_VAL_INT_PLUS_NANO.

Signed-off-by: Marc Ferland <marc.ferland@sonatest.com>
---
Change in v2:
 - Improve commit message as suggested by Jonathan.

 drivers/iio/dac/ad5592r-base.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/iio/dac/ad5592r-base.c b/drivers/iio/dac/ad5592r-base.c
index 076bc9ecfb49..4763402dbcd6 100644
--- a/drivers/iio/dac/ad5592r-base.c
+++ b/drivers/iio/dac/ad5592r-base.c
@@ -415,7 +415,7 @@ static int ad5592r_read_raw(struct iio_dev *iio_dev,
 			s64 tmp = *val * (3767897513LL / 25LL);
 			*val = div_s64_rem(tmp, 1000000000LL, val2);
 
-			return IIO_VAL_INT_PLUS_MICRO;
+			return IIO_VAL_INT_PLUS_NANO;
 		}
 
 		mutex_lock(&st->lock);

base-commit: 98369dccd2f8e16bf4c6621053af7aa4821dcf8e
-- 
2.34.1

Re: [PATCH v2] iio: dac: ad5592r: fix temperature channel scaling value

[Jonathan Cameron]

On Wed,  1 May 2024 11:05:54 -0400
marc.ferland@gmail.com wrote:

> From: Marc Ferland <marc.ferland@sonatest.com>
> 
> The scale value for the temperature channel is (assuming Vref=2.5 and
> the datasheet):
> 
>     376.7897513
> 
> When calculating both val and val2 for the temperature scale we
> use (3767897513/25) and multiply it by Vref (here I assume 2500mV) to
> obtain:
> 
>   2500 * (3767897513/25) ==> 376789751300
> 
> Finally we divide with remainder by 10^9 to get:
> 
>     val = 376
>     val2 = 789751300
> 
> However, we return IIO_VAL_INT_PLUS_MICRO (should have been NANO) as
> the scale type. So when converting the raw temperature value to the
> 'processed' temperature value we will get (assuming raw=810,
> offset=-753):
> 
>     processed = (raw + offset) * scale_val
>               = (810 + -753) * 376
> 	      = 21432
> 
>     processed += div((raw + offset) * scale_val2, 10^6)
>               += div((810 + -753) * 789751300, 10^6)
> 	      += 45015
>     ==> 66447
>     ==> 66.4 Celcius  
> 
> instead of the expected 21.5 Celsius.
> 
> Fix this issue by changing IIO_VAL_INT_PLUS_MICRO to
> IIO_VAL_INT_PLUS_NANO.
> 
> Signed-off-by: Marc Ferland <marc.ferland@sonatest.com>
+CC Nuno.

Seems obviously correct to me so I'll apply it to the fixes-togreg
branch of iio.git.  I've been wrong before though so ideally would like
one of the ADI team to sanity check this.

I'll add the fixes tag as well and mark it for stable. This goes all the
way back.

Fixes: 56ca9db862bf ("iio: dac: Add support for the AD5592R/AD5593R ADCs/DACs")


> ---
> Change in v2:
>  - Improve commit message as suggested by Jonathan.
> 
>  drivers/iio/dac/ad5592r-base.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/iio/dac/ad5592r-base.c b/drivers/iio/dac/ad5592r-base.c
> index 076bc9ecfb49..4763402dbcd6 100644
> --- a/drivers/iio/dac/ad5592r-base.c
> +++ b/drivers/iio/dac/ad5592r-base.c
> @@ -415,7 +415,7 @@ static int ad5592r_read_raw(struct iio_dev *iio_dev,
>  			s64 tmp = *val * (3767897513LL / 25LL);
>  			*val = div_s64_rem(tmp, 1000000000LL, val2);
>  
> -			return IIO_VAL_INT_PLUS_MICRO;
> +			return IIO_VAL_INT_PLUS_NANO;
>  		}
>  
>  		mutex_lock(&st->lock);
> 
> base-commit: 98369dccd2f8e16bf4c6621053af7aa4821dcf8e