Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > Developer Center > Large number formatting->loss of precision

Large number formatting->loss of precision
Thread Tools
Mac Enthusiast
Join Date: Sep 2000
Location: Vermont, USA
Status: Offline
Reply With Quote
May 8, 2003, 10:52 AM
 
Hi,

I have a table view which uses an NSNumberFormatter for one of its table columns. The formatting works great where it displays numbers like this: 1,333.23

However, if the user double clicks the cell in the table view and enters a large number say, 199,999.01 the penny is dropped so that it then looks like 199,999.00

I tried changing the formatters rounding behavior, but still no luck.

I have a model class which takes and returns a float for receiving edited values and returning values to the table. Do I need to change my float variable in my model class to a double?

Thanks in advance.
(Last edited by macrophyllum; May 8, 2003 at 01:26 PM. )
     
Dedicated MacNNer
Join Date: Sep 2002
Status: Offline
Reply With Quote
May 9, 2003, 03:44 PM
 
Do I need to change my float variable in my model class to a double?
Quite possibly... IIRC, single precision IEEE format only gives 6 decimal s.f. What does float map to in your project?
     
Mac Enthusiast
Join Date: Sep 2000
Location: Vermont, USA
Status: Offline
Reply With Quote
May 10, 2003, 12:01 AM
 
OK, switching to a double helped. Now it remains accurate a lot better! Thanks.
     
   
Thread Tools
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Top
Privacy Policy
All times are GMT -5. The time now is 03:29 PM.
All contents of these forums © 1995-2011 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.7 © 2000-2011, Jelsoft Enterprises Ltd., Content Relevant URLs by vBSEO 3.3.2