Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [floating-point]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
1answer
64 views

Two conversions of practically the same number giving two different results

To my surprise, I run the following queries in SQL Server 2014 and each returns a different result, as shown in this db<>fiddle... This Select convert(numeric(19,2),convert(float,9....
0
votes
1answer
27 views

Handling MySQL length=22 and decimals=31

I'm using MySQL's C api to determine the maximum possible value that can be stored in a specific field. I have the following table: CREATE TABLE test ( f1 double(4,2), f2 double ) When I ...
10
votes
1answer
389 views

Non-deterministic sum of floats

Let me state the obvious fist: I completely understand that floating point types cannot accurately represent decimal values. This is not about that! Nevertheless, floating point calculations are ...
0
votes
1answer
151 views

How do you create a Geographic Point with MariaDB and MySQL?

PostGIS provides a simple cast to construct a geography type from floating point lat, and long SELECT ST_Point(1,2)::geography; -- standard version (MM) less precision SELECT ST_MakePoint(1,2)::...
0
votes
1answer
308 views

Does MySQL or MariaDB support double literals (double precision)?

MySQL seems to support double precision The FLOAT and DOUBLE types represent approximate numeric data values. MySQL uses four bytes for single-precision values and eight bytes for double-precision ...
2
votes
1answer
35 views

When PostgreSQL query <Value returns =Values

I'm running a query using PostgreSQL 9.6.6 and when querying: SELECT * FROM table WHERE threshold_value < VALUE returns all values = VALUE and not < VALUE as it should. In the same way ...
1
vote
1answer
41 views

How can I find the stored binary value of a DOUBLE value in MySQL?

Given the following table, backed by MySQL 5.7/InnoDB: CREATE TABLE mytable(mydouble DOUBLE) SELECT 0.005 `mydouble`; how can I find what exactly InnoDB is storing on the disk for the value 0.005? ...
1
vote
1answer
4k views

TSQL Convert FLOAT to STRING Truncation/Rounding Issue

SQL Server 2008 R2 Why when converting a FLOAT to a string (Varchar / Nvarchar) using Cast or Convert, does a rounding or truncation occur? Using CAST DECLARE @floatVal FLOAT = 4.76758527755737 ...
0
votes
2answers
3k views

the result of REPLACE and CONVERT is a VARCHAR(8000) - how can I have a VARCHAR(50) as a result instead?

I have many situation where I need to convert big float numbers with many decimal places to a rounded number with 2, or 1 or no decimal places at all. the way I have been doing this is demonstrated ...
3
votes
4answers
1k views

How to store very small values (E-314) in SQL Server database?

I had created table which hold small floating values using below query create table temp_data (val float) But when it gives warning message for very small values (e-314) insert into ...
6
votes
1answer
2k views

Why does SQL Server allow a float(53) value that falls outside the range specified by Microsoft?

I have a particular table with a column data type of float(53), non null. According to the Microsoft Documentation, the smallest value above 0 that can be stored in this data type is is 2.23E-308. ...
2
votes
2answers
4k views

Biggest possible double/float for a column in my MySQL database

I am developing a PHP application in which I am in need to store values bigger than 1 million, but with the flexibility of having floating points. I know the traditional float and double. I am looking ...
3
votes
1answer
528 views

PostgreSQL CAST() giving unexpected results

I can't seem to figure out how this is rounding overflow digits. It is most definitely not any of: Round up if last digit is 5 or higher. Round to even if last digit is 5. Randomly round up or down ...
1
vote
1answer
1k views

MySQL bug - decimal comparison

I'm currently writing unit tests for a stored procedure that calculates the amount of taxes and the tax rate based on the country/province the customer lives in. Somehow, MySQL considers that "14.975"...
1
vote
1answer
350 views

Modulo operator with nonintegers - unexpected results

MySQL 5.5.32, Oracle 11g MySQL returns strange results for queries involving mod() with noninteger moduli. Why is this? I have a column defined thus: +--------+--------------+------+-----+---------+...
1
vote
1answer
194 views

How can I import latitude and longitude data into a database using Mysql Workbench?

I am currently trying to get some latitude and longitude data to import into a database using the data import wizard in mysql workbench. the problem is that the floating point values that get ...
3
votes
1answer
1k views

Storing floating point values for various units of measurement (kg, ml, mm, pl)

I have a table like the following: order_item ------------ serial | id (PK) integer | order_id ???? | quantity integer | unit_of_measure_id (PK) --> ml, l, kg, mg, mm, m, etc. ...
6
votes
1answer
15k views

PostgreSQL and MONEY data type for currency values

For a project, I created a table with a column price MONEY NOT NULL column. And I thought it would handle decimals properly, unlike a floating number (i.e. IEEE rounding issues), but I end up having ...
1
vote
2answers
3k views

MySQL returns floats rounded

> CREATE TABLE test(foo FLOAT); > INSERT INTO test VALUES(1.2899999); > SELECT * FROM test; +------+ | foo | +------+ | 1.29 | +------+ While it's obvious that FLOAT is not a precise data ...
2
votes
0answers
590 views

Schema for user drag and drop ordering [duplicate]

I'm trying to work out the schema for a system where users will be able to order items in an (unlimited length) list by drag and drop. Is there any kind of standard way of storing this ordering ...
5
votes
1answer
1k views

Rounding issues?

Try this: create table test (f float); insert into test values (330.0); commit; select (8 + 330.0/60)::int; --14 select (8 + f/60)::int from test; --14 select (9 + 330.0/60)::int; --15 select (9 + f/...
1
vote
3answers
2k views

How to enforce full integer (no floating point)?

Why is the following allowed INSERT INTO c VALUES (123.2,'abcdefghijklmnopqrstuvwxyzabcdef', 'abcdefghijklmnop'); When table contains CustomerID NUMBER(3,0) -- That's the 123.2 entry In other ...
1
vote
1answer
243 views

How to prepare a date and double for a mysql replace statement?

I have a MySql table with three columns: CU VARCHAR(3) DA DATETIME VA DOUBLE I need to execute several REPLACE into mytable (CU, DA, VA) values("ABC", mydate, myval) statements from Java. mydate is ...
4
votes
3answers
15k views

MySQL conversion from float (10,2) to fixed(10,2)

I have a database that contains financial data stored as floats. Naturally, this can present a real issue at random times (eg, when there is an exact value that cannot be approximated closely enough ...