Dealing with oil resource data across several disparate systems and also trying to make our API's more generic and consistent, we use WCF a lot (web services with windows apps). We've had a couple of instances where we've had to tweak the config settings for either our publishers or our clients to return some large datasets over WCF.
For the most part, I have found the error messages relating to constrained WCF config settings refreshingly to the point. They pretty much pointed me directly to the attribute in the file that I needed to change in order to fix my issue.
For memory related errors on the client, I found that juggling with the maxBufferPoolSize and maxReceivedMessageSize in my binding to help. I moved the maxBufferPoolSize from 9999999 to 99999999, and changed the maxReceivedMessageSize to 350000000. Sometimes after that I would get a timeout error on the client machine (depending on what kind of load the server was under). So I'd have to make the timeout settings on the binding higher.
Obviously it helps when the publisher has the same values for the same attributes on its binding.