Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [floating-point]

The tag has no usage guidance.

24 questions
Filter by
Sorted by
Tagged with
64 views

Two conversions of practically the same number giving two different results

To my surprise, I run the following queries in SQL Server 2014 and each returns a different result, as shown in this db<>fiddle... This Select convert(numeric(19,2),convert(float,9....
27 views

Handling MySQL length=22 and decimals=31

I'm using MySQL's C api to determine the maximum possible value that can be stored in a specific field. I have the following table: CREATE TABLE test ( f1 double(4,2), f2 double ) When I ...
389 views

Non-deterministic sum of floats

Let me state the obvious fist: I completely understand that floating point types cannot accurately represent decimal values. This is not about that! Nevertheless, floating point calculations are ...
151 views

How do you create a Geographic Point with MariaDB and MySQL?

PostGIS provides a simple cast to construct a geography type from floating point lat, and long SELECT ST_Point(1,2)::geography; -- standard version (MM) less precision SELECT ST_MakePoint(1,2)::...
308 views

Does MySQL or MariaDB support double literals (double precision)?

MySQL seems to support double precision The FLOAT and DOUBLE types represent approximate numeric data values. MySQL uses four bytes for single-precision values and eight bytes for double-precision ...
35 views

When PostgreSQL query <Value returns =Values

I'm running a query using PostgreSQL 9.6.6 and when querying: SELECT * FROM table WHERE threshold_value < VALUE returns all values = VALUE and not < VALUE as it should. In the same way ...
41 views

How can I find the stored binary value of a DOUBLE value in MySQL?

Given the following table, backed by MySQL 5.7/InnoDB: CREATE TABLE mytable(mydouble DOUBLE) SELECT 0.005 `mydouble`; how can I find what exactly InnoDB is storing on the disk for the value 0.005? ...
4k views

TSQL Convert FLOAT to STRING Truncation/Rounding Issue

SQL Server 2008 R2 Why when converting a FLOAT to a string (Varchar / Nvarchar) using Cast or Convert, does a rounding or truncation occur? Using CAST DECLARE @floatVal FLOAT = 4.76758527755737 ...
3k views

the result of REPLACE and CONVERT is a VARCHAR(8000) - how can I have a VARCHAR(50) as a result instead?

I have many situation where I need to convert big float numbers with many decimal places to a rounded number with 2, or 1 or no decimal places at all. the way I have been doing this is demonstrated ...
1k views

How to store very small values (E-314) in SQL Server database?

I had created table which hold small floating values using below query create table temp_data (val float) But when it gives warning message for very small values (e-314) insert into ...
2k views

Why does SQL Server allow a float(53) value that falls outside the range specified by Microsoft?

I have a particular table with a column data type of float(53), non null. According to the Microsoft Documentation, the smallest value above 0 that can be stored in this data type is is 2.23E-308. ...
4k views

Biggest possible double/float for a column in my MySQL database

I am developing a PHP application in which I am in need to store values bigger than 1 million, but with the flexibility of having floating points. I know the traditional float and double. I am looking ...
528 views

PostgreSQL CAST() giving unexpected results

I can't seem to figure out how this is rounding overflow digits. It is most definitely not any of: Round up if last digit is 5 or higher. Round to even if last digit is 5. Randomly round up or down ...
1k views

MySQL bug - decimal comparison

I'm currently writing unit tests for a stored procedure that calculates the amount of taxes and the tax rate based on the country/province the customer lives in. Somehow, MySQL considers that "14.975"...
350 views

Modulo operator with nonintegers - unexpected results

MySQL 5.5.32, Oracle 11g MySQL returns strange results for queries involving mod() with noninteger moduli. Why is this? I have a column defined thus: +--------+--------------+------+-----+---------+...
194 views

How can I import latitude and longitude data into a database using Mysql Workbench?

I am currently trying to get some latitude and longitude data to import into a database using the data import wizard in mysql workbench. the problem is that the floating point values that get ...
1k views

Storing floating point values for various units of measurement (kg, ml, mm, pl)

I have a table like the following: order_item ------------ serial | id (PK) integer | order_id ???? | quantity integer | unit_of_measure_id (PK) --> ml, l, kg, mg, mm, m, etc. ...
15k views

PostgreSQL and MONEY data type for currency values

For a project, I created a table with a column price MONEY NOT NULL column. And I thought it would handle decimals properly, unlike a floating number (i.e. IEEE rounding issues), but I end up having ...
3k views

MySQL returns floats rounded

> CREATE TABLE test(foo FLOAT); > INSERT INTO test VALUES(1.2899999); > SELECT * FROM test; +------+ | foo | +------+ | 1.29 | +------+ While it's obvious that FLOAT is not a precise data ...
590 views

Schema for user drag and drop ordering [duplicate]

I'm trying to work out the schema for a system where users will be able to order items in an (unlimited length) list by drag and drop. Is there any kind of standard way of storing this ordering ...
1k views

Rounding issues?

Try this: create table test (f float); insert into test values (330.0); commit; select (8 + 330.0/60)::int; --14 select (8 + f/60)::int from test; --14 select (9 + 330.0/60)::int; --15 select (9 + f/...
2k views

How to enforce full integer (no floating point)?

Why is the following allowed INSERT INTO c VALUES (123.2,'abcdefghijklmnopqrstuvwxyzabcdef', 'abcdefghijklmnop'); When table contains CustomerID NUMBER(3,0) -- That's the 123.2 entry In other ...