As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [floating-point]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3 votes
2 answers
609 views

What data type is better to use for a PostgreSQL to store number with a floating point and why?

I have a Laravel application that needs to store minimum and maximum weight. Currently, we are using integer values to store data but I need to modify it to store data values with a floating point. ...
Taras's user avatar
  • 147
0 votes
1 answer
39 views

Does Sqlserver store number of relevant decimals in a float field [closed]

I have a table in sql server containing a float field. This field displays a different number of decimals from row to row. I know for a fact that a float field CAN NOT contain the exact value 3173.77, ...
Bob's user avatar
  • 1
1 vote
1 answer
64 views

How to store decimal numbers in MS Access

I tried to store decimal number, but couldn't. It's Office 2019. I tried both number types, and different "format". One format had some zero's show up after a comma, but could not enter ...
Valter Ekholm's user avatar
1 vote
1 answer
30 views

Selecting a floating value yields incorrect results

Consider the following. CREATE TABLE IF NOT EXISTS `docs` ( `id` int(6) unsigned NOT NULL, `qty` float NOT NULL, `price` float NOT NULL, `rebate` float not null, PRIMARY KEY (`id`) ) DEFAULT ...
FiddlingAway's user avatar
1 vote
1 answer
136 views

How to retain same FLOAT value after converting it to a VARCHAR in Azure Synapse

Consider the following example: DROP TABLE dbo.TEST_FLOAT; CREATE TABLE dbo.TEST_FLOAT ( AMOUNT FLOAT ); INSERT INTO dbo.TEST_FLOAT (AMOUNT) VALUES (11546.25); SELECT AMOUNT, CAST(AMOUNT AS ...
Lauren_G's user avatar
9 votes
1 answer
2k views

Why does the cast from double precision to numeric round to 15 significant digits?

The cast from double precision (float8) to numeric rounds to 15 significant decimal digits, thereby losing information. Clearly, more precision is possible. The cast to bigint (for values within its ...
Erwin Brandstetter's user avatar
0 votes
1 answer
694 views

wrong value when storing big float value

i got a weird problem using mariadb. CREATE DATABASE testdb; USE testdb; CREATE TABLE testing (afloat FLOAT NULL); INSERT INTO testing VALUES(123456789.0); SELECT * FROM testing; But instead of ...
JeffreyH's user avatar
3 votes
2 answers
16k views

Why does SQL Server convert floats to scientific notation?

I came across some weird behavior: While passing a float value into a varchar column, the values are getting converted from integers into scientific notation, and it's that scientific notation that ...
James's user avatar
  • 2,602
5 votes
1 answer
6k views

Whats the difference between float and numeric in Postgresql?

Reading the Postgresql docs about the numeric data types leads me to this question: why do I get these unexpected results with the data types Float (SQL standard) and Numeric in Postgresql? For ...
turtle's user avatar
  • 151
0 votes
1 answer
184 views

HeidiSQL rounds down double values upon insert...select

I have the following table: CREATE TEMPORARY TABLE `new_aps` ( `MinuteStart` datetime NOT NULL, `ProxyListID` bigint(20) NOT NULL, `SuccessCount` double(22,0) NOT NULL, `UnknownCount` double(...
Serge Rogatch's user avatar
2 votes
2 answers
2k views

Crazy automatic float rounding in MySQL 5.7 [duplicate]

My exact MySQL version is 5.7.29-0ubuntu0.18.04.1. And I'm beginning to think that I'm getting crazy... Just try these queries: CREATE TABLE `float_bug` ( `v` FLOAT NOT NULL ) ENGINE=INNODB ...
Stalinko's user avatar
  • 161
0 votes
1 answer
162 views

Convert a numeric column with number eg.(-1222222) to two float column with two decimal places eg (-12222.22)

I know there are different variations of this question asked. But what I am trying to do is that, there is a table with a numeric column (12,0) that has data looking like Units ----- -1222222 0 -19 ...
masksoverflow's user avatar
0 votes
1 answer
72 views

Two conversions of practically the same number giving two different results

To my surprise, I run the following queries in SQL Server 2014 and each returns a different result, as shown in this db<>fiddle... This Select convert(numeric(19,2),convert(float,9....
JohnG's user avatar
  • 1,071
0 votes
1 answer
179 views

Handling MySQL length=22 and decimals=31

I'm using MySQL's C api to determine the maximum possible value that can be stored in a specific field. I have the following table: CREATE TABLE test ( f1 double(4,2), f2 double ) When I ...
DemCodeLines's user avatar
11 votes
1 answer
2k views

Non-deterministic sum of floats

Let me state the obvious fist: I completely understand that floating point types cannot accurately represent decimal values. This is not about that! Nevertheless, floating point calculations are ...
Heinzi's user avatar
  • 3,145
0 votes
1 answer
1k views

How do you create a Geographic Point with MariaDB and MySQL?

PostGIS provides a simple cast to construct a geography type from floating point lat, and long SELECT ST_Point(1,2)::geography; -- standard version (MM) less precision SELECT ST_MakePoint(1,2)::...
Evan Carroll's user avatar
  • 60.4k
0 votes
1 answer
624 views

Does MySQL or MariaDB support double literals (double precision)?

MySQL seems to support double precision The FLOAT and DOUBLE types represent approximate numeric data values. MySQL uses four bytes for single-precision values and eight bytes for double-precision ...
Evan Carroll's user avatar
  • 60.4k
2 votes
1 answer
61 views

When PostgreSQL query <Value returns =Values

I'm running a query using PostgreSQL 9.6.6 and when querying: SELECT * FROM table WHERE threshold_value < VALUE returns all values = VALUE and not < VALUE as it should. In the same way ...
Diogo Guerra's user avatar
1 vote
1 answer
99 views

How can I find the stored binary value of a DOUBLE value in MySQL?

Given the following table, backed by MySQL 5.7/InnoDB: CREATE TABLE mytable(mydouble DOUBLE) SELECT 0.005 `mydouble`; how can I find what exactly InnoDB is storing on the disk for the value 0.005? ...
Marcus's user avatar
  • 336
1 vote
1 answer
15k views

TSQL Convert FLOAT to STRING Truncation/Rounding Issue

SQL Server 2008 R2 Why when converting a FLOAT to a string (Varchar / Nvarchar) using Cast or Convert, does a rounding or truncation occur? Using CAST DECLARE @floatVal FLOAT = 4.76758527755737 ...
Mazhar's user avatar
  • 783
0 votes
2 answers
5k views

the result of REPLACE and CONVERT is a VARCHAR(8000) - how can I have a VARCHAR(50) as a result instead?

I have many situation where I need to convert big float numbers with many decimal places to a rounded number with 2, or 1 or no decimal places at all. the way I have been doing this is demonstrated ...
Marcello Miorelli's user avatar
3 votes
4 answers
3k views

How to store very small values (E-314) in SQL Server database?

I had created table which hold small floating values using below query create table temp_data (val float) But when it gives warning message for very small values (e-314) insert into ...
Pravin's user avatar
  • 143
6 votes
1 answer
3k views

Why does SQL Server allow a float(53) value that falls outside the range specified by Microsoft?

I have a particular table with a column data type of float(53), non null. According to the Microsoft Documentation, the smallest value above 0 that can be stored in this data type is is 2.23E-308. ...
Brad's user avatar
  • 174
3 votes
2 answers
15k views

Biggest possible double/float for a column in my MySQL database

I am developing a PHP application in which I am in need to store values bigger than 1 million, but with the flexibility of having floating points. I know the traditional float and double. I am looking ...
idelara's user avatar
  • 133
3 votes
1 answer
1k views

PostgreSQL CAST() giving unexpected results

I can't seem to figure out how this is rounding overflow digits. It is most definitely not any of: Round up if last digit is 5 or higher. Round to even if last digit is 5. Randomly round up or down ...
Jim's user avatar
  • 133
1 vote
1 answer
3k views

MySQL bug - decimal comparison

I'm currently writing unit tests for a stored procedure that calculates the amount of taxes and the tax rate based on the country/province the customer lives in. Somehow, MySQL considers that "14.975"...
Jonathan Parent Lévesque's user avatar
3 votes
1 answer
793 views

Modulo operator with nonintegers - unexpected results

MySQL 5.5.32, Oracle 11g MySQL returns strange results for queries involving mod() with noninteger moduli. Why is this? I have a column defined thus: +--------+--------------+------+-----+---------+...
WAF's user avatar
  • 329
1 vote
1 answer
349 views

How can I import latitude and longitude data into a database using Mysql Workbench?

I am currently trying to get some latitude and longitude data to import into a database using the data import wizard in mysql workbench. the problem is that the floating point values that get ...
Duck Puncher's user avatar
5 votes
1 answer
2k views

Storing floating point values for various units of measurement (kg, ml, mm, pl)

I have a table like the following: order_item ------------ serial | id (PK) integer | order_id ???? | quantity integer | unit_of_measure_id (PK) --> ml, l, kg, mg, mm, m, etc. ...
José María's user avatar
11 votes
1 answer
27k views

PostgreSQL and MONEY data type for currency values

For a project, I created a table with a column price MONEY NOT NULL column. And I thought it would handle decimals properly, unlike a floating number (i.e. IEEE rounding issues), but I end up having ...
Yanick Rochon's user avatar
1 vote
2 answers
4k views

MySQL returns floats rounded

> CREATE TABLE test(foo FLOAT); > INSERT INTO test VALUES(1.2899999); > SELECT * FROM test; +------+ | foo | +------+ | 1.29 | +------+ While it's obvious that FLOAT is not a precise data ...
MaxSem's user avatar
  • 111
2 votes
0 answers
2k views

Schema for user drag and drop ordering [duplicate]

I'm trying to work out the schema for a system where users will be able to order items in an (unlimited length) list by drag and drop. Is there any kind of standard way of storing this ordering ...
wombley's user avatar
  • 21
5 votes
1 answer
2k views

Rounding issues?

Try this: create table test (f float); insert into test values (330.0); commit; select (8 + 330.0/60)::int; --14 select (8 + f/60)::int from test; --14 select (9 + 330.0/60)::int; --15 select (9 + f/...
kev's user avatar
  • 395
1 vote
3 answers
2k views

How to enforce full integer (no floating point)?

Why is the following allowed INSERT INTO c VALUES (123.2,'abcdefghijklmnopqrstuvwxyzabcdef', 'abcdefghijklmnop'); When table contains CustomerID NUMBER(3,0) -- That's the 123.2 entry In other ...
JAM's user avatar
  • 135
1 vote
1 answer
304 views

How to prepare a date and double for a mysql replace statement?

I have a MySql table with three columns: CU VARCHAR(3) DA DATETIME VA DOUBLE I need to execute several REPLACE into mytable (CU, DA, VA) values("ABC", mydate, myval) statements from Java. mydate is ...
Jérôme Verstrynge's user avatar
4 votes
3 answers
20k views

MySQL conversion from float (10,2) to fixed(10,2)

I have a database that contains financial data stored as floats. Naturally, this can present a real issue at random times (eg, when there is an exact value that cannot be approximated closely enough ...
Bryan Agee's user avatar