[wings_rfs]

For a while now I've been using MemoryStick to display my Mac's memory usage. After several days of uptime I see nearly all of my 1GB of ram being used, mostly in the Inactive catagory but a substantial amount in Active as well. And, sometiimes if I've been using Azureus, Toast or Thoth (all of which gobble memory like crazy) I can have up to 3 swap files in use. OK, nothing wrong with that I guess - I understand that is the way its supposed to work. But lately I'm only seeing 1 swap file no matter what, and Active at about 20-25% of my total, and Inactive at around 25-50% of the total. It grows at times but it soon settles back down to these amounts. If I had to guess, this is the way I would have thought it would work (releasing memory if it hasn't been accessed in a while). And then I realized.... it started this when I upgraded to 10.3.5.

Has anyone else observed this with the new update?

[Big Mac]

I cannot tell you whether or not the behavior you're seeing is different from that of previous Panther iterations. But, the fact that you see a lot of Inactive memory being used is totally normal. As you may know, OS X caches recently used memory in the hopes of using it again in the future - Inactive memory. The OS is designed to use most available memory and leave very little of it unused. Moreover, swap files are normal because they are devoted to applications which often reserve large amounts of memory. The thing to pay attention to is your pageouts - if it's significantly larger than your pageins, you need more RAM. I bookmarked a wonderful explanation of all this on my G5, but I'm on my iBook right now. Later on I may update this post with the link.

[Thorin]

Originally posted by Big Mac:
The thing to pay attention to is your pageouts - if it's significantly larger than your pageins, you need more RAM. I bookmarked a wonderful explanation of all this on my G5, but I'm on my iBook right now. Later on I may update this post with the link.

Please do! I only have 512MB in my 12" powerbook, and I suspect that upgrading to 768MB would give me a bit of a performance boost, but it's stretching my finanaces a bit to do it at the moment. I'd be really interested to read that article and get a better idea as to how much of a performance increase I'd see with just an extra 256MB of RAM.

[tkmd]

I've had concerns about MM in OSX. It seemed that even with 1.25 GB of memory I would still drop to ~10-20MB as a free number. After, feeling like I had been ripped off (after all I have 1.25GB memory surely there is enough to around and plenty to be left free). After about two weeks of keeping the -top command open in the UNIX shell I observed a couple of interesting facts. First there are two thing that gobble up memory; browsers( especially Firefox with its memory leak) and anything that runs JAVA (limewire, Aquasition, Azureous etc). Closing these programs when done really kicks back alot of memory to the free category. Second, programs that grind away for hours crunching data (DVD->.avi conversion) Apparently OSX allows these programs to take as much memory as needed.

But even though I was converting video and my memory dropped to 20 MB free I didn't noticed a slow down because although I paged out temporarily, I recovered memory from the inactive pile immediately. I guess my point is that one really needs at least 768MB of mem in OSX especially if your running JAVA apps and rely on the feel of the OS during times of HD thrashing and heavy use. If you cant feel a difference in the responsiveness of the OS - Memory management is probably doing its job regaurdless of the spread of the numbers in Memory stick.

[someone_else]

Originally posted by tkmd:
I've had concerns about MM in OSX. It seemed that even with 1.25 GB of memory I would still drop to ~10-20MB as a free number. After, feeling like I had been ripped off (after all I have 1.25GB memory surely there is enough to around and plenty to be left free).

Wow. That's a pretty asinine statement. Why would you feel ripped off because all of your memory is being put to use? I'd feel ripped off if it wasn't being used. I mean what reason is there to have all of that memory to just have it sit there and do nothing?

Free memory means next to jack squat in OS X. OS X is all about the pageouts. Those should be your guide for if you need more memory or not.

[Dale Sorel]

Originally posted by someone_else:
That's a pretty asinine statement.

Having a bad day

[suthercd]

Watch pageouts increase as you download data- Thoth/bitTorrent clients especially. Also will create swapfiles. That reflects the way the software is written--- poor memory handling routines.

C-

[Graymalkin]

OSX uses memory efficiently. If you think you can do better, download the Darwin source and write your own memory manager. In OSX memory is handled virtually. This means every process is given its very own virtual memory space which limited by 32-bit addressing is up to 4GB (G5's can offer more virtual memory to processes). Thus every program is led to believe by the kernel that it is the only thing running on the computer. This virtual memory space is divided up into "pages" and whenever a program needs a bit of memory the kernel gives it a new page.

Pages have three lists, Active, Inactive, and Free. Active pages are ones that contain data and have been recently accessed, Inactive pages also contain valid data but haven't been accessed recently and may be released whenever the kernel sees fit, Free pages have yet to be assigned to a process. When the amount of Free pages drops below a particular threshold (determined by the amount of physical memory you have) the Inactive pages are scanned for particular things. If Inactive pages have been accessed recently they're stuck in the Active list, if they haven't been accessed recently but still contain valid data they're moved to the backing store (written to disk), if they haven't been modified and isn't in the Wired (memory that can't be paged out to disk) list it is stolen and the page is put on the Free list. When a program explicitly frees memory it is also added to the Free list. The designers of OSX's aren't the idiots some people insinuate that they are.

Some programs use a lot more memory than others, not necessarily due to "memory leaks" but because of the way they are designed. Safari for instance uses a metric buttload of memory. It does this because it caches damn near everything that it touches in memory. Ever wonder why Safari is so quick to move backwards and forwards between pages? It keeps the pre-rendered versions of those pages in a cache whenever possible. This isn't necessarily a memory leak, it is Safari's way of keeping your web browsing speedy.

BitTorrent clients store a lot of data in RAM not because they have a "memory leak" but that is the way they're designed. BitTorrent downloads files into a cache and serves them back out as they're downloaded. That is how BitTorrent is designed to work. BT needs to be able to serve whatever segments of the file it has out to incoming clients, it can't predictively know what parts of the file it will get and can thus serve back out so it can't easily write those files to disk and not keep them in memory. Firefox uses a lot of memory because it is a self contained rendering system. Firefox renders the entire contents of itself with its own rendering engine. The only parts of the app really using native OSX components are the container windows.

If you're not a programmer don't throw the term "memory leak" around as you're probably not 100% sure of what it means. Using a large amount of memory for caches is not a memory leak. It might be a bad design in some situations but it isn't a true "memory leak". Complaining about OSX's memory manager is equally silly if you don't really know what you're saying. OSX will use whatever memory it can get its hands on but for efficiency reasons will page out to disk no matter how much physical RAM you have. You might have a whole ton of of pageouts with 1GB of RAM, that isn't necessarily a bad thing. OSX is just trying to use as much of that 1GB for active running processes as it can.

[msuper69]

Originally posted by Graymalkin:
OSX uses memory efficiently. If you think you can do better, download the Darwin source and write your own memory manager.
...

Well put. Finally somebody who has a good understanding of the concept (and OS X implementation) of virtual memory. Your explanation should be made a sticky post and people should be referred there whenever one of those imbecile "OS X memory management sucks" threads gets posted.

[yukon]

Yeah, everytime I read a thread that mentions memory usage, it always goes to misconceptions, then more than one explination, and that derails the thread. I'd like to see a sticky about this and any other topics like it. BTW, wanting some or even substantial physical memory free at all times isn't assinine, it's beneficial when you open a new application etc, some free memory makes new tasks go faster...>100mb free I'm sure you see that advantage lost by then to systems that cache more data.

From all the OS X versions I've used, I've found that memory management behavior does change, Apple works to tweak such things all the time it would seem (isn't that awesome?). It seemed to me to be inefficient as well in Panther, I see more swap files than ever, and I wasn't certain if that's a good thing (massively paging out inactive memory rather than purging? disk media is slow, it might be faster to recalculate or reretrieve data sometimes)....if there are fewer swap files in the new version (I haven't updated), it's possible Apple tweaked the memory management. Can anyone check the CVS log or something of Darwin/XNU? ISTR hearing that was updated.

[someone_else]

Originally posted by yukon:
BTW, wanting some or even substantial physical memory free at all times isn't assinine, it's beneficial when you open a new application etc, some free memory makes new tasks go faster...>100mb free I'm sure you see that advantage lost by then to systems that cache more data.

Read Graymalkin's post again. He hit the nail on the head. It's called the Inactive list. If a new app is launched, then the pages will be pulled from the Inactive list, but until that time there should be no "free memory", it should be used to cache data that was used sometime in the past. You do not want "free memory" (as in the free list). If there is true free memory, it just means the OS isn't caching as much as it could (or you haven't put that much pressure on the system yet).

If you get in the situation that there on no inactive or free pages, and active pages have to start being paged out to free up pages for new apps, then it's time to buy some more memory.

[yukon]

Then OS X is inefficient, I have 10mb of free memory (I've never seen it stay below that for more than a minute). If data has to be taken out of RAM and paged to disk in order to free RAM for a new application, guess what, Safari takes an extra "bounce" or so. The above 10mb is there for a reason, possibly Apple has done it's research and found that 10mb is a good tradeoff between cache and instantly available RAM space, as well (or more importantly) allowing enough time for data to be paged out or discarded so that there isn't a delay, the paging out and then in IO can happen more simultaneously. You seem to believe that pages in the free list still have data in them, which is true as they can be reclaimed before it's overwritten, but overwriting them doesn't require paging out (it's freed, it's probably not wanted anymore. EDIT, there is no speed penalty IOW).

When the amount of Free pages drops below a particular threshold (determined by the amount of physical memory you have) the Inactive pages are scanned for particular things.

I didn't read the mentioned post before, I read about half of it, realized it was to explain memory management (I've only read a few hundred of those) and scanned the rest. Anyway, thread derailed, thanks for focusing on my "BTW" to explain more about memory management.

[Graymalkin]

The kernel will do everything in its power to give memory to an application that needs it at that moment. While sometimes this can lead to swap thrashing (and a tremendous slowdown) it usually works out for the better. The kernel tries to maintain a hunk of pages on the Free list in order to quickly give that you an allocation request by a currently or newly running program but it isn't always able to when there's a lot of active programs. This is why it can be a PITA to run a lot of applications with relatively little RAM. There's a huge difference between the performance of a system (especially laptops) with 256MB of RAM and one with 1GB of RAM.

Having VM files on the hard disk is not necessarily and absolutely a performance killer. When you've got an uptime of weeks your system will maintain quite a few VM files on the disk. These aren't filled with pages that are necessarily needed by anything and won't do anything but tie up some space on your hard drive. Think of how many applications and files you've worked with in the past three weeks (assuming your uptime is three weeks). That is a lot of file caches and allocated memory pages. Finder might have up to 200MB of virtual memory assigned to it with only 40-50MB actively being used at any given time. All of those unused pages are being flagged as Inactive and either kept in physical RAM until they're needed or paged out to disk. Statistically 70% of those inactive pages won't be needed again and will eventually be discarded or just left to sit in a VM file until the system restarts. The number of swapfiles doesn't necessarily mean there's more pageouts between two revisions of Panther, it simply means the pager filled up a single VM file and started another.

The memory manager in OSX has indeed changed but it's become far more efficient than in previous versions. Earlier versions of OSX were designed to run with as little as 64MB of RAM. This predicated that the pager has to be very aggressively tuned in order to have enough free pages in physical memory for active processes to use. Even with 128MB of RAM the pager was still fairly aggressive. Things have improved since the recommended minimum has been declared to be 256MB with Panther. The pager can be far less agressive with its pageouts and as such there's going to be far less disk thrashing. You also have to consider that when Cheetah and Puma were newly released a huge percentage of applications were Carbon ports of Classic apps. These were applications designed around the unified address space of MacOS 7-9 with pre-set memory allocations. These apps were allocating as much memory as the developers expected them to ever use which the kernel was obliged to give them. So instead of an app needing 40MB and getting 40MB it was getting 60MB though it was only using 40MB. A single app could allocate more memory than a system running Puma might even have to give and would absolutely have to pageout to disk.

There's a lot more to memory management than I think some people think there is. It's a very complex system that is hard to understand. If you really don't understand it or only have a tenuous understanding try reading up on the issue rather than complaining about nonissues in the OS. I'm not about to complain that a NASCAR driver didn't win a race because the pit crew ran too rich a mixture through the fuel injector. It's entirely likely they did everything right and the driver lost the race for some other reason. Besides, I'm hardly an expert on the tuning of racecars.

[suthercd]

This reference has been a good overview.

Craig

[yukon]

Thanks Graymalkin, you obviously had a better understanding of the memory management then I did when OS X was less developed, I just knew things had changed by the free and swap behavior I was seeing. I have to say though, in Panther, I did notice a very large increase in pageouts (and pageins), possibly I should check out what's doing that a bit more closely (or get the full gig of ram, PC100 prices never drop anymore...). I had assumed there'd still be a speed penalty when cached data is loaded off the HD when there's so much swap space and the data could just be reloaded from disk (or in whatever case). I certainly never thought "Oh, what idiots, I don't need that much swap!", it just confused the logic

To be honest, I really don't care much about swap, I only check it every few months to make sure I still have enough memory. I have a seperate swap partition (hey, I had a new unpartitioned drive, lots of space, why not?) on a seperate drive, it's over a gig in size, I don't see any effects unless an OS X update has reverted it to the small-ish OS X system drive partition.

[someone_else]

Originally posted by yukon:
Then OS X is inefficient, I have 10mb of free memory (I've never seen it stay below that for more than a minute). If data has to be taken out of RAM and paged to disk in order to free RAM for a new application, guess what, Safari takes an extra "bounce" or so. The above 10mb is there for a reason, possibly Apple has done it's research and found that 10mb is a good tradeoff between cache and instantly available RAM space...

No, OS X is very effiient. That 10MB (or more depending on your mem. size) always free is for the kernel, it has nothing to do with "quick access" or "instantly available" pages. Accessing a page from the inactive list is just as fast (as long as it's not dirty) as accessing one from the free list.

However, when the kernel allocates memory for itself, it cannot take the chance of having to pull a page from disk. Therefore, it always reserves a small amount of free pages for itself, if that reserve is ever dipped into, then it will page out inactive (and if necessary active) pages to put them back on the free list. This way, the kernel always has some memory available for itself. So, yes you could start seeing a page thrash even if there is a small amount of free memory, and now you know why. If this does happen, then it's time to buy some more memory.

[yukon]

Originally posted by Someone_Else-
...but until that time there should be no "free memory", it should be used to cache data that was used sometime in the past. You do not want "free memory" (as in the free list). If there is true free memory, it just means the OS isn't caching as much as it could (or you haven't put that much pressure on the system yet).

There should be no free memory.

Originally posted by Someone_Else-
Therefore, it always reserves a small amount of free pages for itself, if that reserve is ever dipped into, then it will page out inactive (and if necessary active) pages to put them back on the free list.

There should be free memory.

You can see the reason I took exception (my "Then OS X is inefficient" line was to call attention to the fact that there was free space, that any free space _shouldn't_ simply "be used to cache data"). RE kernel needing space free, of course, without that there'd be a lockup if there wasn't anything free or immediatly freeable and the kernel needed it. But that's not the only purpose of having some free memory.

I didn't code the memory manager, but I make sure to check what I say if I'm hazy on it before I say it.
http://www.google.com/search?q=%22fr...threshold+disk

[wings_rfs]

In my original post I said....
"But lately I'm only seeing 1 swap file no matter what, and Active at about 20-25% of my total, and Inactive at around 25-50% of the total. It grows at times but it soon settles back down to these amounts.
Has anyone else observed this with the new update?"

Well... has anyone????

[suthercd]

As a step in the boot process, /private/var/vm directory is established if needed and old swap files are removed if they exist. Dynamic_pager is started, which extablishes the profile for the swapfile(s) and sets the parameters for swapfile creation/deletion. On my machine there is one swapfile created after boot by default.

HTH
Craig

[Zim]

For something that was so "perfect" before, 10.3.5 is a LOT better.

I had huge swapfile problems and drive thrashing (leading to pronounced system performance degradation vs. 10.2.x) under 10.3.0->10.3.4, but so far under 10.3.5 I'm using ~2 swapfiles LESS than before (I think the reclamation was less than good before).

My system is significantly "snappier" again

Mike