Integer number from Lua calculation needed

Help creating logic scripts for Air Manager Instruments

Moderators: russ, Ralph

Message
Author
Detlef
Posts: 304
Joined: Mon Nov 16, 2020 9:43 am
Location: Bavaria

Integer number from Lua calculation needed

#1 Post by Detlef »

Hi,

in an instrument script I want to control the brightness of a text color. For that I have the variable gColorPrint.

From time to time I see this error in the logfile:
"error in dataref callback: logic.lua:266: bad argument #2 to 'format' (number has no integer representation)"

It happens in this line in my lua instrument script:

gColorPrint = string.format("#%02X%02X%02XFF",
toint(255*brightness_lines),
toint(255*brightness_lines),
toint(255*brightness_lines))

toint() is a function I found on the web that should make sure that only integer numbers are returned.:

Code: Select all

function toint(n)
  -- needed because math.floor() returns floats
    local s = tostring(n)
    local i, j = s:find('%.')
    if i then
        return tonumber(s:sub(1, i-1))
    else
        return n
    end
end
I know it is a somewhat lua expert question. What I need is a method to make sure I get an integer value from a calulation like 234 * 0.665467
Before I have tried without toint() which also did not work in all situations.

Thank you for any help
Detlef

User avatar
Corjan
Posts: 2936
Joined: Thu Nov 19, 2015 9:04 am

Re: Integer number from Lua calculation needed

#2 Post by Corjan »

Hi,

I think lua uses math.floor or math.ceil.

Corjan

User avatar
Sling
Posts: 5237
Joined: Mon Sep 11, 2017 2:37 pm
Contact:

Re: Integer number from Lua calculation needed

#3 Post by Sling »

You can also use var_round(var) if you prefer rounding to floor or ceil. If you omit the decimal place argument it defaults to 0.

User avatar
Ralph
Posts: 7878
Joined: Tue Oct 27, 2015 7:02 pm
Location: De Steeg
Contact:

Re: Integer number from Lua calculation needed

#4 Post by Ralph »

You can check the variable type with print( math.type(value) )
But I also believe that Corjan is correct.

User avatar
Keith Baxter
Posts: 4674
Joined: Wed Dec 20, 2017 11:00 am
Location: Botswana

Re: Integer number from Lua calculation needed

#5 Post by Keith Baxter »

Hi,

What Corjan and the guys have said is correct but there are things to consider depending on the application.

Using what is mentioned can have it's issue. math.floor() and var_round() or var_format() produce the same. math.cell() is different.
math.ceil , math.floor
Return the integer no greater than or no less than the given value (even for negatives).
> = math.floor(0.5)
0
> = math.ceil(0.5)
1
> = math.floor(-0.5)
-1
> = math.ceil(-0.5)
-0

A value of 4.5 will return a 5 and a value of 5.4 will also return a 5. What this means is that the whole number changes at .5 and not at .0.
This has an impact when working with the likes clocks. You will find that a floor() of round() minute value will change when the second value is 1/2 minute.

To overcome this it is best to use string,format() and string.sub()

Code: Select all

rl_disp_digit=string.format("%02.0f",string.sub(cron_s,1,2))
Some "time related" datarefs 10th's are not decimal. In other words change after the 5.9th 10th's. (59 seconds change to 0 seconds)
Also when using radio frequencies when do you want a FM 49.7 frequency to change? Do you want it to change when it is 49.75 or 49.70.

Just something to be aware of and consider.
In the OP case probably not important but for others just be aware of the different methods.


Keith

EDIT: Yes formatting is clumsy, but accurate for your specific application. Rem to convert the value to the correct type using to.string() OR to.number()
AMD RYZEN 9 5950X CPU, Corsair H80I cooler, ASUS TUF GAMING B550-PLUS AMD Ryzen Mother Board,  32Gb ram Corsair Vengeance 3000Mh, MSI GTX960 4G graphics card 

User avatar
Keith Baxter
Posts: 4674
Joined: Wed Dec 20, 2017 11:00 am
Location: Botswana

Re: Integer number from Lua calculation needed

#6 Post by Keith Baxter »

Hi

I reread your initial post. I assume you are using 8 digit hex code to control the opacity.

Using math.floor( ) will create errors as values will be outside of 0..255 scope.

You can use var_cap() for this.
Willing to assist on the discord should you want.

Keith
AMD RYZEN 9 5950X CPU, Corsair H80I cooler, ASUS TUF GAMING B550-PLUS AMD Ryzen Mother Board,  32Gb ram Corsair Vengeance 3000Mh, MSI GTX960 4G graphics card 

Detlef
Posts: 304
Joined: Mon Nov 16, 2020 9:43 am
Location: Bavaria

Re: Integer number from Lua calculation needed

#7 Post by Detlef »

Hi again,

thank you all for your expertise. In my case string.format() did not accept the return value of math.floor() as an integer. I used string.format() with %x formatting to produce a hexadecimal number.

I have changed my code now to:
gColorPrint = string.format("#%02X%02X%02XFF",
toint(math.floor(gNightR*brightness_lines)),
toint(math.floor(gNightG*brightness_lines)),
toint(math.floor(gNightB*brightness_lines)))

I think I did it that way before and the errors did not pop up anymore. I will see. It is somewhat uncertain when an error happens because of the daylight changing my brightness value.

Thank you
Detlef

User avatar
Keith Baxter
Posts: 4674
Joined: Wed Dec 20, 2017 11:00 am
Location: Botswana

Re: Integer number from Lua calculation needed

#8 Post by Keith Baxter »

Hi,

You could also add a error check something like.

If xyz ~= nill then ...

I think you want to convert a decimal input into a Hex input
Also how are you converting a dec input to a hex input ? do you * by .255 or some other math

Keith
AMD RYZEN 9 5950X CPU, Corsair H80I cooler, ASUS TUF GAMING B550-PLUS AMD Ryzen Mother Board,  32Gb ram Corsair Vengeance 3000Mh, MSI GTX960 4G graphics card 

User avatar
jph
Posts: 2846
Joined: Fri Apr 10, 2020 12:50 pm
Location: Somewhere over the rainbow..

Re: Integer number from Lua calculation needed

#9 Post by jph »

Keith Baxter wrote: Tue Nov 22, 2022 11:25 am
Using math.floor( ) will create errors as values will be outside of 0..255 scope.

You can use var_cap() for this.
Keith
That should never be an issue as the 'brightness' multiplier per colour would presumably be <= 1
That isn't a problem.
Joe. CISSP, MSc.

User avatar
Ralph
Posts: 7878
Joined: Tue Oct 27, 2015 7:02 pm
Location: De Steeg
Contact:

Re: Integer number from Lua calculation needed

#10 Post by Ralph »

In canvas you can use float values on a fill. So 0,0,0 is black and 1,1,1 is white. I'm not sure if it works on text, you can try, saves you some trouble.

Post Reply